White Paper: Application Guidance PER-005

The MRO Subject Matter Expert Team is an industry stakeholder group which includes subject matter experts from MRO member organizations in various technical areas. Any materials, guidance, and views from stakeholder groups are meant to be helpful to industry participants; but should not be considered approved or endorsed by MRO staff or its board of directors unless specified.

APPLICATION GUIDE PER-005

Report Date: December 15, 2011

Authored by

Richard Cobb, Midwest ISO Steve Zimmerman, Wisconsin Public Service Will Behnke, Alliant Energy Ron Bauer, American Transmission Company Chris Bellefy, Minnesota Power Robert Kieborz, Nebraska Public Power District Carla Fitzpatrick, Omaha Public Power District John Pettingill, Xcel Energy Patrick Matson, Madison Gas and Electric

P a g e | 2


Disclaimer The Midwest Reliability Organization (MRO) Standards Committee (SC) is committed to providing training and non-binding guidance to industry stakeholders regarding existing and emerging Reliability Standards. Any materials, including presentations, were developed through the Standards Committee by Subject Matter Experts from member organizations within the MRO. Subject Matter Experts in the field of System Operator Training were brought together in 2011, to prepare a guide for complying with NERC Reliability Standard PER-005 Training. Participants include representatives from Balancing Authorities, Reliability Coordinators, and Transmission Operators.

PER-005 Subject Matter Expert –Application Guide Development Team John Pettingill, Chair Rich Cobb, Vice Chair Chris Bellefy Xcel Energy MISO Energy Minnesota Power Pat Matson Carla Fitzpatrick Robert Kieborz Madison Gas and Electric Omaha Public Power District Nebraska Public Power Ron Bauer Steve Zimmerman Will Behnke American Transmission Company Wisconsin Public Service Alliant Energy The materials have been reviewed by MRO staff and provide reasonable application guidance for the standard(s) addressed. Ultimately, demonstrating compliance depends on a number of factors including the precise language of the standard, the specific facts and circumstances, and quality of evidence. These documents may be reproduced or distributed to any person or entity only in its entirety.

P a g e | 3


Acknowledgement This publication was developed by a team of Subject Matter Experts (SME) from MRO member organizations within the MRO footprint. The development of SME teams is an ongoing effort to produce unified application guides for MRO and its Registered Entities. The PER-005 SME team chair, John H. Pettingill from Xcel Energy, wishes to acknowledge and thank those who dedicated efforts and contributed significantly to this publication. The MRO and the MRO Standards Committee, and their organizational affiliations include:

Midwest Reliability Organization James Burley, Vice President William Smith, Manager Compliance, Mitigation and Risk Assessment, Standards and Program Performance and Standards

Russ Mountjoy, Manager Jennifer Matz Compliance Mitigation & Standards Administrator

MRO Standards Committee Joe Knight, Chair David Rudolph, Vice Chair Dave Action Great River Energy Basin Electric Power Cooperative Alliant Energy

Wayne Guttormson Beth Lemke Lloyd Linke Saskatchewan Power Wisconsin Public Service Western Area Power Administration Mike Moltane Tim Noeldner Andrew Pusztai ITC Holdings WPPI Energy American Transmission Company Gerry Steffens Mike Garton Robert Thompson Rochester Public Utilities Dominion Resources Services Xcel Energy

P a g e | 4


TABLE OF CONTENTS INTRODUCTION .......................................................................................................................... 5

OVERVIEW ................................................................................................................................... 5

METHODOLOGY ......................................................................................................................... 6

R1. ............................................................................................................................................... 6

R1.1. ............................................................................................................................................ 9

R1.1.1 ........................................................................................................................................ 12

R1.2. .......................................................................................................................................... 14

R1.3. .......................................................................................................................................... 16

R1.4. .......................................................................................................................................... 17

R2 .............................................................................................................................................. 18

R2.1. .......................................................................................................................................... 20

R3. ............................................................................................................................................. 21

R3.1. .......................................................................................................................................... 23

APPENDIX A: REFERENCES ................................................................................................... 25

APPENDIX B: GLOSSARY ....................................................................................................... 27

APPENDIX C: ACRONYM LIST .............................................................................................. 29

APPENDIX D: Reliability-Related Task Selection Filter ........................................................... 30

APPENDIX E: Reliability-Related Task Modification Filter - Example .................................... 31

APPENDIX F: Systematic Approach to Training – Roadmap Examples ................................... 32

P a g e | 5


INTRODUCTION NERC Reliability Standard PER-005-1 is a new training standard for system personnel that will replace NERC Reliability Standard PER-002. The purpose and intent of this standard is to ensure that System Operators are competent to perform real-time, reliability-related tasks on the North American Bulk Electric System (BES). NERC Reliability Standard PER-005-1 is applicable to the following functional entities: Reliability Coordinators, Balancing Authority and Transmission Operator.

OVERVIEW A systematic approach will guide trainers in determining which training solutions to retain, refine, and/or supplement. There are other systematic approaches to training that should be acceptable. Many of our existing qualification programs already use performance-based training, which does not necessarily have to be discarded to implement PER-005. NERC is likely to accept any performance-based systematic approach that is widely used in industry or government (e.g. Technology Training System of Richard Swanson of the University of Minnesota). The goals of this SME Project include the following: • Clarify training terminology

• Explain how requirements are interrelated

• Describe processes and outcomes that can be used for evidence of compliance.

P a g e | 6


METHODOLOGY This section contains suggested methodologies to meet the requirements of NERC Standard PER-005. These methods represent the intended practices of the members of the PER-005 SME Team.

FERC Comments on Systematic Approach to Training FERC provides insight into how they view a systematic approach to training. See the Federal Register, Vol. 75, No. 227. Friday, November 26, 2010, page 72664 to 72679. FERC also addresses the question regarding the specifications from PER-002 that are not explicit in PER-005, such as instructor qualifications and plans for initial and continuing training. Read Section C. Systematic Approach to Training, Commission Determination numbers, 25, 31, and 38. Also check Section D. Training Expectations for Each Job Function/Tailored Training NOPR Proposal for more background.

[Commission Determination] 25. A systematic approach to training is a widely-accepted methodology that ensures training is efficiently and effectively conducted and is directly related to the needs of the position in question. To achieve training results, the objectives of a systematic approach to training include: management and administration of training and qualification programs; development and qualification of training staff; trainee entry-level requirements; determination of training program content; design and development of training programs; conduct of training; trainee examinations and evaluations; and training program evaluation.

[Commission Determination] 31. Currently effective Reliability Standard PER–002–0, Requirement R3.2 explicitly mandates that ‘‘the training program must include a plan for the initial and continuing training of Transmission Operators and Balancing Authorities operating personnel.’’ In the NOPR, the Commission sought an explanation from NERC, and comment from the general public, whether continuing training is an enforceable requirement under proposed Reliability Standard PER–005–1 and whether this requirement is clear or should be more explicit. [Commission Determination] 38. Based on the comments received, the Commission concludes that the current requirement for each training program (that training staff must be identified and that such staff must be competent in both knowledge of system operations and instructional capabilities) is inherent in any systematic approach to training that a registered entity would use to meet this requirement, and thus is an enforceable component of Requirement R1 under the proposed standard.

R1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall use a systematic approach to training to establish a training program for the BES company-specific reliability-related tasks performed by its System Operators and shall implement the program.

P a g e | 7


PER-005 may be revised in the future to specifically address FERC’s concerns so you may want to address these FERC concerns when initially implementing PER-005.

Systematic Approach to Training ADDIE is an acronym for Analysis, Design, Development, Implementation and Evaluation. The Military originated ADDIE to create and deliver performance-based job training. While there is wide acceptance of the definition of the ADDIE approach, some authors may show differences in the locations of some specific outcomes. For example, some authors place the creation of lesson plans in the Design phase, while others place it in the Development phase. Outcomes for each phase of ADDIE are listed as follows: • Analysis: identifies the “requirements” • Design: identifies the “specifications” • Development: creates the “training solution” • Implementation: is the “actual delivery” • Evaluation: is the “continuous improvement”

Implementation The simplest and most direct method to implement a systematic approach to training is to complete the phases of ADDIE in sequence. 1. Analysis: Identify what instruction is needed to qualify new employees and what instruction

is needed to maintain or improve performance. Job and Task Analysis, including identification of tasks, are both performed at this phase.

2. Design: Determine the characteristics of the training which will meet the needs identified in the Analysis phase. Identify the competencies and objectives needed for each task. Determine how instruction will be accomplished (learning strategies). The assessments that come out of the Design phase enable trainers to demonstrate that instruction has actually been accomplished (assessments of learners).

3. Development: Create the materials and media (e.g. training guides, OJT manuals, slides, simulation exercises) driven by the design specifications from the Design phase.

4. Implementation: Create a plan for the scheduling of classroom preparation, sign-in of participants, implementing the lesson plan, student engagement, student assessment, making evaluations available, and recordkeeping. Deliver the training, provide evaluations, and record the results.

5. Evaluation: Create an ongoing process for identifying the changes needed in training setting, training activities, scheduling, course content, materials, etc.

P a g e | 8


Documentation

Evidence Evidence consists of two major types: (1) the training policies and procedures that govern the training program, and (2) the outcomes of the systematic approach itself. These training policies and procedures also provide the documentation for the “Training Program”. The evidence you create in meeting the other PER-005 requirements R1.1 through R3.1 will help provide most of the documentation you need for R1, Systematic Approach to Training. That is, if you adequately document your training processes and outcomes. Your processes are documented with policies, procedures, plans, process maps, etc. Your outcomes are documented with training records, schedules, planning documents, curriculum, etc. There are a few additional forms of evidence not included in the evidence provided in the other requirements. These are listed in the compliance roadmap example provided in the Appendix F of the Application Guide. In general, the following process and outcomes will be needed.

Training Policies and Procedures: • Job Descriptions and Duties of System Operators.

• Initial Qualification Plan – how a new employee achieves qualification.

• Continuing Education Plan – how continuing training is identified and assigned.

• Records management – how completion of training is recorded and retrieved.

• Evaluation – of courses and programs.

• Training Scheduling – when Operators can obtain training.

• Instructor Qualification – qualifications for the designated trainer and for subject matter

experts. Documentation could also include how technical and instructional qualifications are maintained.

• Capability verification – trainers assess the capabilities of each System Operator. This can include qualifications of the Task Verifiers, the verification process, and how trainers address incumbents who fail a scorecard.

• Needs Analysis – how new training needs are identified. This includes how trainers

determine that a task has changed or new tasks have been added.

P a g e | 9


Outcomes: • Task lists

• Task details or steps for each task

• Job Performance Measures (scorecards) for each task

• Learning Objectives (associated with each task; associated with each training solution)

• Materials and Media (examples are training guides, CBT, Video, and PowerPoint slides)

• Lesson Plans (for example, as documented in the NERC Individual Learning Applications)

• Instructor list (with qualifications)

• Assessment Instruments (examples are scorecards, exercises, and examinations)

• Completed Evaluations

• Completed Capability Verification instruments

• Transcripts of Student Records

This requirement is part of the Analysis phase of the ADDIE Systematic Approach to Training described above. A task is an identifiable, observable and measurable piece of a job that serves as a unit of work. When a simple task statement is combined with job steps, necessary equipment, conditions, and performance standards (e.g. no errors, completed in a timely manner) it can be used as a Job Performance Measure (JPM). These JPMs in turn, can be used to assess Operator Capability. Work can be organized into duties, tasks, and steps. Each duty has an associated group of tasks. Each task has an associated group of steps. And each task or step has associated learning objectives that must be mastered in order to perform that task or step. If the analysis has identified less than ten units of work for a job position, it may be including duties that need to be broken down further. If the analysis has several hundred units of work for single job function, it may have included steps which are part of tasks.

R1.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall create a list of BES company-specific reliability-related tasks performed by the System Operators.

P a g e | 10


Identifying Job Related Task A major challenge is converting work processes into a manageable number of tasks. There needs to be sufficient granularity to construct meaningful task assessments in order to verify the employee is qualified to perform the task. Yet, there should not be too many tasks as to create an administrative burden. Technical job positions can usually be organized into 40-100 tasks. If a company has employees who are cross-trained in order to rotate between different functions, then obviously the total for that job position would be significantly larger. In contrast, with the advent of the MISO, many balancing and interchange tasks have been assumed by MISO Operations. Consequently, the number of system operator tasks performed for this function may be less than the rule of thumb mentioned above. In addition, since we are looking just at real-time tasks and just at reliability-related tasks, the total task list will be even smaller. Note: Writing tasks is not an exact science. Different trainers may come up with a slightly different, yet workable, set of tasks for identical circumstances. Tasks can be written as simple actions, describing only the action and object of the action or they can be more complex including the action, object of action and purpose or result. Whatever style is used the tasks should be written following pre-established guidelines to maintain uniformity of task statements. When tasks are written so they involve multiple actions or compound statements it may be necessary to create multiple tasks. The task analysis process should not simply rely on “generic” lists provided by external sources. However, these lists are a useful resource. Note that R1.1 states “company-specific reliability-related tasks”. Trainers must therefore analyze the work processes, internal and external procedures, regulatory environment, and topology unique to their system. Use operator interviews, regulatory standards, operating procedures, surveys, and job-shadowing to ensure that company-specific tasks have been identified.

Reliability-Related Task One of the early outcomes in the job and task analysis process is a preliminary list of tasks. These generally come from observation on the job and/or interviews. To classify this list of tasks as reliability-related, determine if the system operator task is:

• Linked to the NERC Compliance Registry (BA, TOP, or RC)

• Specified or implied in a NERC Standard

• Linked to a relationship described in the NERC Functional Model

P a g e | 11


• Specified in some other NERC document

• Specified in RTO/ISO process that impacts BES

Real-Time Task Determine if a task is performed in real-time or the “present”, as your company defines it. In Appendix D, there is an example of a process chart for classifying an existing task as being subject to PER-005; the task is both reliability-related and occurs in real-time. This process chart can be expanded upon to meet regional and individual company variations.

Task Survey The final company-specific task list should be reviewed by subject matter experts and/or incumbent operators to identify the frequency, importance, and difficulty of each task. This review could be accomplished through interviews or surveys. This analysis also helps identify which tasks are covered in initial training and in continuing training. For example, system restoration tasks would usually be scored as low frequency, high importance, and high difficulty. This type of score would necessitate continuing training on a periodic basis. Below is an example of a task scaling survey form using task statement examples from NERC.

Figure 1.0: Task Scaling Survey

(See DOE HDBK-1076-94 on how to use task survey results).

P a g e | 12


Relationship between Task Analysis and Developing an Assessment The task analysis that creates the list of tasks provides information for the development of job performance measures (JPM), OJT Guides, and Examinations. These assessments will be used in constructing the Capability Verification Assessments used in Requirement R2.

Maintaining a Task List Placing the task analysis data into a computer database or spreadsheet is recommended. This repository will allow the data to be sorted and organized to be more efficiently used later in the systematic approach and for audit purposes.

Reporting capability All the information developed in the task analysis can be documented in a job analysis report. The report can list the positions of people conducting the analysis, the methods and process used and the results. The report can be incorporated into the overall training program. This documentation is useful in program continuity and to explain the methodology in an internal or external audit.

Documenting Task List The measurement cited in the NERC Standard provides guidance as to the evidence needed for compliance.

M1.1. Each Reliability Coordinator, Balancing Authority, and Transmission Operator shall have available for inspection its company-specific reliability-related task list, with the date of the last review and/or revision, as specified in R1.1.

In addition to the task list and revision tracking, be prepared to share your process for identifying new and modified tasks. See R.1.1.1 below. R1.1.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall update its list of BES company-specific reliability-related tasks performed by its System Operators each calendar year to identify new or modified tasks for inclusion in training.

Updating the task list. This requirement has two parts. You will need to note the modification of an existing task or creation of a new task as they occur. You will also need to train to these new tasks within six months per requirement 2.1. In addition, we believe you should review all tasks annually, not just those that are new or modified.

Identifying New Task and/or Modifying an Existing Task. To identify changes in the task list, monitor changes in the following: regulatory environment, work conditions, tools work processes, and desired work outcomes for each unit of work. The

P a g e | 13


ongoing operational procedural reviews that most companies conduct are a major source for identifying changes in tasks. Many companies have a periodic procedure review process for both internal and MISO procedures. In reviewing and updating these procedures, look for possible changes in the task list. If a new System Operator activity does not appear to fit into an existing task, run this through the task selection process described in section R1.1. To determine if an existing task, subject to PER-005, has been modified: 1. Has any process change occurred, even a very small change? This might include conditions,

tools, number and sequence of steps, hand-offs with others, external procedures, or criteria for successful performance. If the answer is yes, then continue with the questions below. If no discernible changes have occurred in the existing tasks, and yet you are aware of process changes, then determine if a new task was created. Submit this proposed new task to the Task Filter identified earlier and represented graphically in Appendix C.

2. Is the process change more than administrative in nature? If the answer is yes, then continue with the next question. If the answer is no, then the task has not been sufficiently modified to warrant additional training beyond what is planned for that task. This step allows simple changes to be screened out.

3. Will the process change affect “how” the task is performed? If the answer is yes, then continue with the questions below. If the answer is no, then the task has not been sufficiently modified to warrant additional training beyond what is planned for that task.

4. Will the process change affect the “difficulty or complexity” of the task? If the answer is yes, then continue with the questions below. If the answer is no, then the task has not been sufficiently modified to warrant additional training beyond what is planned for that task.

5. If the answer to all the above questions is “Yes”, then o Document changes to the task (e.g. tools, steps, references, criteria, objectives, etc.).

o Validate the changes to this task

o Update your task list (per R1.1.1.) if the wording in the task has changed.

o Create the necessary training.

o Deliver training within six months.

P a g e | 14


Validating Updated Task List Once you have identified new or modified tasks, you can check the validity using Subject Matter Experts, Supervisor reviews, System Operator interviews and surveys, and/or an advisory committee. Document the process and the outcomes for updating tasks. Develop a procedure on the updating process: criteria for a new task, criteria for a modified task, who validates, etc. Retain the outcomes of the updating process. These may include tracking forms, task spreadsheets, operator interviews, surveys, training advisory committee minutes, etc.

Design Phase The Design phase of ADDIE determines the characteristics of the training which can meet the needs identified in the Analysis phase. The Design phase identifies the learning objectives needed for each task which drive the selection of training activities and materials. The Design phase determines how instruction will be accomplished (learning strategies). The learner assessments (OJT Scorecards, examinations) that come out of the Design phase enable trainers to document that instruction has actually been accomplished. Learning objectives refer to the observable student performance used to measure learning. For each task, there are learning objectives that must be mastered in order to perform that task. Learning objectives drive the types of learning strategies (classroom versus simulator) and the specifications for training materials and media. Objectives can be written so they include a condition, action and criterion. An example would be: “given a set of data, the student will compute a balancing authority’s ACE, without error”. (Reference: DOE-HDBK-1200-97). The selection of the action or verb used in a learning objective is critical. These must align with the type and level of learning. Select verbs that align with NERC’s philosophy as reflected in their Continuing Education program. (Reference: NERC’s, Guide to Writing Learning Objectives, for NERC’s definitions and recommended usage of many common learning objective verbs.) The Relationship between learning objectives and tasks The learning objectives must align with the type of task. For example, troubleshooting tasks can be linked with higher level cognitive learning objectives learning objectives are incorporated into lesson plans, NERC Individual Learning Applications, Course Syllabi, etc. The outputs from the design phase will be used in the development phase.

R1.2. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall design and develop learning objectives and training materials based on the task list created in R1.1.

P a g e | 15


The Development phase of the ADDIE process addresses the creation of learning activities, training materials, training methods, and lesson plans. Learning activities can be defined as “during instruction, the planned opportunity for students to produce, practice, or perform the task, skill, or knowledge to be mastered”. Training materials (handouts, learning guides, slides, exercises) must support the learning objectives. Training methods must also align with learning objectives and support the selected learning activities. Here is an example of a task detail sheet which lists the learning objectives associated with a task.

Figure 1.2: Detail Task List with Learning Objectives

Retrieval of Associated Learning Objectives for a Given Task We recommend that trainers have this ability, whether this is performed manually or using a database or spreadsheet. A trainer should also be able to retrieve all the associated training materials that are linked to a task by its learning objectives.

Methodology for choosing “Continuing Training” Topics There are several methods for explaining the selection of “continuing training” topics. Some topics selected are due to regulatory requirements. For example, in NERC EOP-005, R6 states:

P a g e | 16


“Each Transmission Operator and Balancing Authority shall train its operating personnel in the implementation of the restoration plan. Such training shall include simulated exercises, if practicable.”

Another method uses the results of the Task DIF Survey. This survey scores each task by its difficulty, importance, and frequency. Generally, the guidelines for using DIF surveys indicate which tasks need continuous or refresher training. This recurring cycle will depend upon the needs of each company. A third method for selecting continuing training topics is incorporating industry events (e.g. islanding, blackouts, etc.) of significance to reliability.

Development Phase The Development phase of ADDIE determines what materials & media are used (e.g. training guides, OJT manuals, slides, simulation exercises, etc.). It finalizes how learning transfer is delivered (actual delivery such as classroom, CBT, or OJT). The outcomes include the materials and media that are needed. The measurement cited in the NERC Standard gives us limited guidance as to the evidence needed for compliance.

M1.2. Each Reliability Coordinator, Balancing Authority, and Transmission Operator shall have available for inspection its learning objectives and training materials, as specified in R1.2.

For a specific task, trainers should be able to retrieve its associated learning objectives, training plans, and its performance assessment (e.g. capability verification instrument). For a specific task, trainers should be prepared to query the associated training materials linked to the task by its learning objectives. In addition, trainers should be prepared to retrieve a specific training course and show the following: learning objectives, associated tasks, learning assessment, and training materials.

Deliver Phase “Deliver” represents the Implementation Phase of ADDIE. The outcome of the successful Implementation phase is the creation of competent employees. Implementation involves the actual delivery of the training. Elements may include scheduling of participants, classroom preparation, sign-in of participants, pre-assessment, student engagement, student assessment, making evaluations available, and recordkeeping. The measurement cited in the NERC Standard gives us some guidance as to the evidence needed for compliance.

R1.3. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall deliver the training established in R1.2.

P a g e | 17


M1.3. Each Reliability Coordinator, Balancing Authority, and Transmission Operator shall have available for inspection System Operator training records showing the names of the people trained, the title of the training delivered and the dates of delivery to show that it delivered the training, as specified in R1.3.

Document the process and the outcomes. Training policies and procedures associated with training delivery can include: instructor qualifications, course identification system, document revision and control, testing and retesting, and recordkeeping. Consider creating a policy for internal instructors, external instructors, and OJT evaluators. Determine minimum technical qualifications and training needed for these roles. Consider a policy or procedure on using subject matter experts as adjunct instructors; determine whether to require minimum instructor skills of adjunct instructors or to have a designated trainer present when the adjunct instructor is delivering training.

The following records are useful in demonstrating compliance: training schedules, course descriptions, lesson plans, attendance sheets, examination results, completed job performance measures (scorecards), instructor resume, evaluation surveys and training policies and procedures. Consider creating a reporting system to minimize the burden of record-keeping systems. Reporting elements can include generating reports for the Operators, to Management, to NERC, and to the company Learning Management Systems.

Evaluation Phase This requirement represents the Evaluation Phase of the ADDIE Systematic Approach to Training. Evaluation is an ongoing process for identifying the changes needed in training setting, training activities, scheduling, course content, materials, etc. Most trainers use course evaluations to gather feedback from participants. In fact, those trainers that are CEH providers must provide evaluations to the participants. These individual course evaluations can be aggregated to identify changes. In addition, input can be gathered regarding the overall program, both qualification and continuing training. This input can be obtained using surveys, interviews, and/or advisory groups. It is also important to show how this feedback is incorporated into course and program changes.

R1.4. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall conduct an annual evaluation of the training program established in R1, to identify any needed changes to the training program and shall implement the changes identified.

P a g e | 18


Documentation needed in evaluation. The measurement cited in the NERC Standard gives us some guidance as to the evidence needed for compliance.

M1.4 Each Reliability Coordinator, Balancing Authority, and Transmission Operator shall have available for inspection evidence (such as instructor observations, trainee feedback, supervisor feedback, course evaluations, learning assessments, or internal audit results) that it performed an annual training program evaluation, as specified in R1.4.

Course feedback can be aggregated if the forms use a ratings scale. In addition, record where changes were made based on the input received. Documentation can be found in revisions to training procedures, course tracking forms, course design, and program descriptions. R2. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall verify each of its System Operator’s capabilities to perform each assigned task identified in R1.1 at least one time.

“Verify…capabilities” in the Systematic Approach to Training “Capability Verification” is part of the implementation phase. It is a type of assessment created to verify that System Operators are competent to perform the reliability-related tasks. Identify training settings and/or on the job activities that will allow trainers to assess an Operator’s performance of a task. Normal operating tasks can usually be assessed on the job. Emergency activities will usually be assessed in a simulated or training environment. As stated in the Standard, evidence can be documents such as training records showing successful completion of tasks with the employee name and date; supervisor check sheets showing the employee name, date, and task completed; or the results of learning assessments. Job Performance Measures (JPM) is sometimes known as Scorecards. The Design and Development phases should have produced scorecards that identify the acceptable level of performance. These JPMs can include the following information:

• task(s) to be assessed

• task steps

• tools, conditions, references needed if applicable

P a g e | 19


• standards of satisfactory performance for critical steps

• name of evaluator

• name of Operator

• signature for verification

Each step of the JPM should be observed or assessed. If a step is omitted, a dated and initialed written explanation should be made on the JPM. All steps of a JPM should be performed to the stated standard. See below for an example of a JPM for the task of “evacuate control center in the event of emergency.”

Figure 2.0: Job Performance Scorecard

The measure in the NERC Standards gives us some guidance.

M2. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall have available for inspection evidence to show that it verified that each of its System Operators is capable of performing each assigned task identified in R1.1, as specified in R2. This evidence can be documents such as training records showing successful completion of tasks with the employee name and date; supervisor check sheets showing the employee name, date, and task completed; or the results of learning assessments.”

Document the process and the outcomes for Capability Verification. Document the person who is authorized to assess Operators on-the-job and who is authorized to assess Operators in a

P a g e | 20


learning environment. Document the process for developing and updating the Assessments. Be sure to have a process for incumbent system operators who fail a Capability Verification assessment. Since Emergency activities are difficult to verify on the job, trainers will often rely on Learning Assessments. Be prepared to prove that the learning assessment addresses the capabilities you wish to verify. Consider relying on simulation or table-top exercises that can act as proxies for the tasks. Documentation for simulations and table-top exercises includes performance scorecard, logs, observation checklist, etc.

Determining New or Modified Tasks See the methodology described previously under requirement, R1.1.1. It describes how to determine that an existing task has changed sufficiently to require a new verification. Criteria for analyzing potential task modification can include: tool changes (such as a new EMS application), procedure changes, change in performance standards, and change in process hand-offs, etc. Take a conservative approach: when in doubt, consider it a modification that warrants new verification by Operator Assessment. See Appendix E for an example of a Task Filter to determine if circumstances have created a new task or modified an existing task.

The Six Month Time Limit Trainers will make a judgment on whether to conduct training in advance or within the six month time limit for training. Factors in making this decision include the impact of the change on safety, reliability, and organizational performance. For the official six month count down, use the date when the operator has to perform the task in a different manner. That could be the date of the software change; effective date of a new procedure, etc. Methods of Verifying Capabilities See the methodology described previously under requirement, R2. It describes how to determine that an existing task has changed sufficiently to require a new capability verification assessment. Use the same documentation used to meet requirement R1.1.1. (Update task list) and requirement R2 (verify capabilities).

R2.1. Within six months of a modification of the BES company-specific reliability-related tasks, each Reliability Coordinator, Balancing Authority and Transmission Operator shall verify each of its System Operator’s capabilities to perform the new or modified tasks.

P a g e | 21


R3. At least every 12 months each Reliability Coordinator, Balancing Authority and Transmission Operator shall provide each of its System Operators with at least 32 hours of emergency operations training applicable to its organization that reflects emergency operations topics, which includes system restoration using drills, exercises or other training required to maintain qualified personnel. As to the interpretation of “every 12 months” most companies use the calendar year while some use the anniversary date of initial qualification. Most interpretations are that companies should continue to follow the same practice they followed under PER-002, R4. Companies should define and be consistent in their methodology for determining “every 12 months”. Useful guidance is provided in the Midwest Reliability Organization newsletter, published for May/June 2011. www.midwestreliability.org/06_news/newsletters/2011/Newsletter_May_2011.pdf

For those Reliability Standards where the definition of the term “annual” is not included, each registered entity must define and document the definition of “annual.” A single definition may be used for all of the Reliability Standards requiring a definition or a registered entity may choose different definitions for each such Reliability Standards, and therefore have multiple definitions. Regardless of whether a single definition of “annual” is used or multiple definitions of “annual” are used, each registered entity must document the definition of “annual” for each Reliability Standard. According to CAN-0010, the documentation “must demonstrate that the required activity was conducted at least once in every calendar year.” In addition, NERC provided two definitions of “annual” which can be used. Annual can be defined as follows: (1) within a calendar year, with the calendar year beginning on January 1 and ending on December 31 (Calendar year) or (2) within a rolling twelve month period (Rolling 12 Month) therefore requiring that the activity be done once every twelve month period. NERC notes that while the Calendar year definition could be met by performing a compliance activity in January Year 1 and in December Year 2 with twenty three months in between the compliance activities, best practices dictate registered entities conduct “annual” activities within 15 months of the previous activity. MRO agrees, but recognizes that in some cases a gap of more than 15 months may be justified. For example, for training, the time frame between compliance activities may be extended if higher quality training can be implemented rather than completing it within 15 months of the prior training. In setting the definitions for annual it is important to keep in mind the reliability objective, which is to “ensure that entities perform a particular task on a regular basis, with an established maximum interval between the occasions when the task is performed.”

http://www.midwestreliability.org/06_news/newsletters/2011/Newsletter_May_2011.pdf

P a g e | 22


Topics for “Emergency Operations” Training See Attachment A. Recommended Operating Training Topics from the Personnel Subcommittee dated March 2, 2004. It states “…to provide our system operators with hands-on training that is effectively provided by system simulations and drills, along with general study of the principles of interconnected systems operations that relate to recognizing and responding to system emergencies”. This list is also contained in the NERC PER-005-1 Reference Document.

1. Recognition and Response to System Emergencies a. Emergency drills and responses b. Communication tools, protocols, coordination c. Operating from backup control centers d. System operations during unstudied situations e. System Protection f. Geomagnetic disturbances weather impacts on system operations g. System Monitoring – voltage, equipment loading h. Real-time contingency analysis i. Offline system analysis tools j. Monitoring backup plans k. Sabotage, physical, and cyber threats and responses

2. Operating Policies Related to Emergency Operations

a. NERC policies, appendices, standards that deal with emergency operations b. Regional reliability operating policies c. Sub-regional policies and procedures d. ISO/RTO policies and procedures

3. Power System Restoration Philosophy and Practices

a. Blackstart b. Interconnection of islands – building islands c. Load shedding – automatic (under-frequency and under-voltage) and manual d. Load restoration philosophies

4. Interconnected Power System Operations

a. Operations coordination b. Special protections systems c. Special operating guides d. Voltage and reactive control, including responding to eminent voltage collapse e. Concepts of Interconnection Reliability Operating Limits versus System f. Operating Limits g. DC tie operations and procedures during system emergencies h. Thermal and dynamic limits i. Unscheduled flow mitigation − congestion management j. Local and regional line loading procedures k. Radial load and generation operations and procedures

P a g e | 23


l. Tie line operations m. E-tagging and Interchange Scheduling n. Generating unit operating characteristics and limits, especially regarding reactive o. capabilities and the relationship between real and reactive output

5. Technologies and Tools a. Forecasting tools b. Power system study tools c. Interchange Distribution Calculator

6. Market Operations as They Relate to Emergency Operations

a. Market rules b. Locational Marginal Pricing c. Transmission rights d. OASIS e. Tariffs f. Fuel management g. Real-time, hour-ahead and day-ahead tools

The measurement cited in the NERC Standard gives us some guidance as to the evidence needed for compliance.

M3. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall have available for inspection training records that provide evidence that each System Operator has obtained 32 hours of emergency operations training, as specified in R3.

Evidence can include the training records (rosters, drill scenarios, assessments, evaluations) associated with this training. R3.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator that has operational authority or control over Facilities with established IROLs or has established operating guides or protection systems to mitigate IROL violations shall provide each System Operator with emergency operations training using simulation technology such as a simulator, virtual technology, or other technology that replicates the operational behavior of the BES during normal and emergency conditions.

Simulator Modeling Requirements The requirement states that the technology “replicates the operational behavior of the BES”. The measurement cited in the NERC Standard gives us some guidance as to the evidence needed for compliance.

P a g e | 24


M3.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall have available for inspection training records that provide evidence that each System Operator received emergency operations training using simulation technology, as specified in R3.1.

As with R3, the evidence can include training records (rosters, drill scenarios, learning objectives, exercises, assessments, evaluations) associated with this training.

P a g e | 25


APPENDIX A: REFERENCES LAWS, POLICIES, DIRECTIVES, INSTRUCTIONS, STANDARDS AND GUIDELINES

1. B. Kirwan and L.K. Ainsworth. A Guide to Task Analysis. New York, New York: CRC Press, 1992.

2. Department of Energy. DOE Handbook, Guide to Good Practices for Developing

Learning Objectives. DOE-HDBK-1200-97. Washington, DC: 1997.

3. Department of Energy. DOE Handbook, Guide to Good Practices for On-the-Job Training. DOE-HDBK-1206-98. Washington,

4. Department of Energy. DOE Handbook, Guide to Good Practices for the Design,

Development, and Implementation of Examinations. DOE-HDBK-1205-97. Washington, DC: 1994.

5. Department of Energy. DOE Handbook, Guide to Good Practices Evaluation Instruments Examples. DOE-HDBK-1210-97. Washington, DC: 1994.

6. Department of Energy. DOE Handbook, Systematic Approach to Training. DOE-

HDBK-1078-94. Washington, DC: 1994.

7. Department of Energy. DOE Handbook, Table-Top Job Analysis. DOE-HDBK-1076-94. U.S. Department of Energy, Washington, D.C. 20585

8. Department of Energy. DOE Handbook, Table-Top Training Programs Design. DOE-

HDBK-1086-95. Washington, DC: 1995.

9. Diane Walter. Training on the Job. Danvers, MA: ASTD Press, 2001

10. Federal Register. Volume 75, Number 227. Friday, November 26, 2010. Rules and Regulations. Department of Energy. Federal Energy Regulatory Commission. 18 CFR Part 40 [Docket No. RM09–25–000; Order No. 742] System Personnel Training Reliability Standards.

11. MISO Energy. Midwest ISO LBA, TOP, GOP Joint Task List. Carmel, IN: 2010.

12. North American Electric Reliability Corporation. Generic System Operator Task List.

Atlanta, GA: 2006.

13. North American Electric Reliability Corporation. PER-005 System Personnel Training Reference Document. Atlanta, GA: 2008.

14. North American Electric Reliability Corporation. Registration Standards Applicability

List for Coordinated Functional Registration (CFR) JRO00001. Atlanta, GA: 2010.

P a g e | 26


15. North American Electric Reliability Corporation. Reliability Functional Model: Function

Definitions and Functional Entities, Version 5. Functional Model Working Group. Atlanta, GA: 2010.

16. North American Electric Reliability Corporation. Statement of Compliance Registry

Criteria. Atlanta, GA: 2008.

17. North American Electric Reliability Corporation. Guide to Writing Learning Objectives. Atlanta, GA: 2007.

18. Patricia Pulliam Phillips. ASTD Handbook for Measuring and Evaluating Training.

Danvers, MA: 2010.

19. Raynold A. Svenson and Monica J. Rinderer. Training and Development Strategic Plan Workbook. New York, NY: Prentice-Hall, 1992.

20. William Rothwell and H.C. Kazanas. Mastering the Instructional Design Process: A Systematic Approach. San Francisco, CA: Wiley-Pfeiffer, 2003.

P a g e | 27


APPENDIX B: GLOSSARY COMMON TERMS AND DEFINITIONS The following terms and phrases are contained in the PER-005 Standard or affect some of its outcomes. “Adequacy” is the ability of the electric system to supply the aggregate electrical demand and energy requirements of the end-use customers at all times, taking into account scheduled and reasonably expected unscheduled outages of system elements. This term is related to the definition of reliability. Criteria for adequate level of reliability: (1) The System is controlled to stay within acceptable limits during normal conditions. (2) The System performs acceptably after credible Contingencies. (3) The System limits the impact and scope of instability and cascading outages when they occur. (4) The System’s Facilities are protected from unacceptable damage by operating them within Facility Ratings. (5) The System’s integrity can be restored promptly if it is lost. (6) The System has the ability to supply the aggregate electric power and energy requirements of the electricity consumers.

“Bulk Electric System (BES)” meets any of the following requirements:

(1) As defined by the Regional Reliability Organization, the electrical generation resources, transmission lines, interconnections with neighboring systems, and associated equipment, generally operated at voltages of 100 kV or higher. Radial transmission facilities serving only load with one transmission source are generally not included in this definition.

(2) As defined in the Federal Power Act, BES refers to facilities and control systems that are necessary for operating an interconnected electric energy transmission network or any portion thereof; and electric energy from generation facilities needed to maintain transmission system reliability. See NERC Project 2010-17 for updates on the definition of Bulk Electric System. • http://www.nerc.com/filez/standards/Project2010-17_BES.html and • http://www.nerc.com/docs/standards/sar/bes_definition_20110824_clean.pdf

“Competency”, as defined by the David C. McClelland, Department Chair at Harvard University, consists of the knowledge, skills and attitudes required to perform the requirements of the job. “Real-time” refers to present time as opposed to future time (as defined in the NERC Glossary). NERC has no quantifiable measure, but in the NERC Functional Model document, they do categorize three time periods for Balancing Authority and Reliability Coordinator activities: “ahead of time”, “real time”, and “after the hour”. “Reliability-related” is any activity performed by a Certified System Operator or Reliability Coordinator that meets any of the following criteria: (1) associated with the activities identified in the NERC Functional Model, (2) an expressed or implied action in a NERC Reliability

http://www.nerc.com/filez/standards/Project2010-17_BES.html

http://www.nerc.com/docs/standards/sar/bes_definition_20110824_clean.pdf

P a g e | 28


Standard, (3) actions required for operation of the BES with an adequate level of reliability, and (4) actions required by the Reliability Coordinator as documented in its policies and procedures. “System Operator“ refers to an individual at a control center subject to PER-005 (Balancing Authority, Transmission Operator, and Reliability Coordinator); their responsibility is to monitor and control their assigned portion of the bulk electric system in real time. “Systematic approach to training” refers to the methodology for identifying training needs and creating training solutions. The most common systematic approach is ADDIE. ADDIE is an acronym for Analysis, Design, Development, Implementation and Evaluation. ADDIE is the most common methodology for creating and delivering performance-based job training “Task” is a discrete unit of work. It is an action to contribute a specified end result to the accomplishment of an organizational objective. It has an identifiable beginning and end which is a measurable component of a specific job. “Training program” is a plan and/or system of training solutions (courses, on-the-job training, assessments, etc.) for the goal of preparing and maintaining the performance of employees.

P a g e | 29


APPENDIX C: ACRONYM LIST ADDIE: Analysis, Design, Development, Implementation and Evaluation. ADDIE is a

commonly used systematic approach to training.

BES: Bulk Electric System. CBT: Computer-based training. CEH: Continuing Education Hours. Unit of measure for completion of training that can be

applied to the renewal of individual System Operator NERC Certification. DIF: Difficulty, Importance, and Frequency. A DIF survey is often performed during Task

Analysis to scale each task as to these three

DOE: Department of Energy. ILA: Individual Learning Application. The format used by NERC Continuing Education

Hours Providers to request NERC to approve a training course for designated Continuing Education Hours.

ISO: Independent System Operator.

JPM: Job Performance Measure. MISO: Midwest Independent Transmission System Operator. MRO: Midwest Reliability Organization. NERC: North American Electric Reliability Corporation. OJT: On the job training. RRE: Regional Reliability Entity. RTO: Regional Transmission Organization. SAT: Systematic Approach to Training.

P a g e | 30


APPENDIX D: Reliability-Related Task Selection Filter

Task not subject to PER-005: but, continue with ADDIE training process

for that task.Task is subject to PER-005.

Action specified in RTO/ISO process that impacts

BES?

Action linked to a relationship in Functional

Model?

YES

NO

YES

NO

NO

System Operator action linked to NERC

Compliance Registry (BA, TOP, RC)

Reliability action occurs in Real Time?

Action specified or implied in associated NERC Standards?

P a g e | 31


APPENDIX E: Reliability-Related Task Modification Filter - Example

Insert task into the Task Selection Filter.

Change affects how task is performed?

Is change more than administrative?

Change addresses an existing PER-005

Task?

Work process or procedural change?

YES

YES

YES

YES

YES

Continue with ADDIE process to have training completed and capability verified within six months per NERC PER-005

NO

NO

NO

Task is not modified per PER-005, but determine if change in training may still be desired due to other considerations uncovered during this filtering.

Change affects difficulty or

complexity of task?NO

P a g e | 32


APPENDIX F: Systematic Approach to Training – Roadmap Examples The first roadmap tracks ADDIE processes and outcomes, and then identifies the relevant PER-005 requirement.

Phases Processes

(policies, procedures, plans, methodology, etc.)

Outcomes (records, reports, etc.)

Relates to PER-005 Requirement

ADDIE Summary document on following a Systematic Approach to Training – refer to other training procedures and policies

N/A R1. Systematic Approach to Training

Analysis Task Analysis Methodology (real-time, reliability-related)

List of Task Statements R1.1 List of BES company-specific reliability-related tasks

Task Analysis Methodology (task performance standards)

List of Task Performance Standards (task details)

R2. verify each of its System Operator’s capabilities to perform each assigned task

Qualification Methodology – how a new employee achieves qualification

Task Scaling (to select tasks for qualification)

R1 Systematic Approach to Training

Continuing Education Methodology – how continuing training is identified and assigned.

Task Scaling (to select tasks for continuing training)

R1 Systematic Approach to Training

Task Updating Methodology - Determining when tasks have been modified or new tasks created

Updated Task List;

Analysis identifying modified or new tasks

R1.1.1 update BES company-specific reliability-related tasks

Design Training Objectives Methodology – developing and using objectives

List of Learning Objectives; Lesson Plans (or ILAs)

R1.2 …design and develop learning objectives and training materials

Develop-ment

Curriculum Standards, templates, or methodology

Curriculum R1.2 …design and develop learning objectives and training materials

Implemen-tation

Training Staff – criteria for qualifications of Instructors and Capability Assessors

List of Instructors and Capability Assessors

R1. Systematic Approach to Training

Recordkeeping Methodology - how completion of training is recorded and retrieved.

Training Records

R1.3 …deliver the training established in R1.2

P a g e | 33


APPENDIX F: Systematic Approach to Training – Roadmap Examples

Operator Scheduling- when Operators obtain training.

Training Schedules, Training Calendar

R1.3 …deliver the training established in R1.2

Phases Processes (policies, procedures, plans, methodology, etc.)

Outcomes (records, reports, etc.)

Relates to PER-005 Requirement

Qualification Methodology – how a new employee achieves qualification; Capability Verification Methodology

Initial Qualification records or Capability Verification Records

R2. verify each of its System Operator’s capabilities to perform each assigned task

Capability Verification Methodology Capability Verification Records

R2.1 Within six months…verify each of its System Operator’s capabilities to perform the new or modified tasks


Training Records (that identify Emergency Operations)

R3. …provide each of its System Operators with at least 32 hours of emergency operations training


Training Records (that identify Emergency Operations); Lesson Plans (or ILAs)

R3.1 …provide each System Operator with emergency operations training using simulation technology

Evaluation Evaluation Methodology Surveys, Advisory meetings, interviews, etc.; Report on how training program has been amended based on evaluation feedback.

R1.4 conduct an annual evaluation

P a g e | 34


APPENDIX F: Systematic Approach to Training – Roadmap Examples Example of a Compliance tracking roadmap (provided by one of the participants)

Legend: I = implementation process/documentation L = Long-term process/documentation B = Both

Documents

White Paper: Application Guidance PER-005