View
224
Download
2
Category
Tags:
Preview:
Citation preview
Good Practice Performance Indicators
Engineering Practices Working Group
Performance Indicator Sub-Group
Engineering Performance Metrics
D. C. Lowe
October 21, 2003
2
Performance Indicator Sub-Group Participants
• Dave Lowe, CH2M HILL Hanford - Chair• Herb Mumford, Bechtel, INEEL• Barb Quivey, LLNL• Tom Monahon, Westinghouse Savannah River• Harry Gullet, Sandia National Laboratories
3
Objectives • Identify a set of “good practice” example PIs from
commercial nuclear and DOE sites grouped into general categories applicable to engineering that could be adopted or modified, as appropriate, for use by engineering groups within the DOE complex.
• Demonstrate that engineering groups are practicing “good engineering.”
• Identify where engineering should focus attention to satisfy customer needs.
• Identify trends in equipment or system performance to focus resources correctly.
• Monitor engineering costs and efficiency.
4
Approach
• Gather input/examples from INPO, commercial nuclear, and DOE sites.
• Identify general categories that engineering groups typically monitor.
• Evaluate input from participating sites/plants corresponding to general categories.
• Provide good practice examples in each category.
5
PI Example Contributors
DOE Sites• Hanford Site (Tank Farms)• Savannah River Site• INEEL
Commercial Nuclear Plants• Columbia Generating Station• Davis Besse• McGuire Nuclear Station• Watts Bar Nuclear Plant• Wolf Creek Nuclear
6
Engineering PI Categories
A. Product Quality/Technical Rigor
B. Safety System Performance
C. Configuration Control
D. Production/Productivity
E. Continuous Improvement
F. Training and Qualification
G. Engineering Cost
7
Product Quality/Technical RigorGood Practice Examples
1. Document Quality– Change package quality– Engineering technical adequacy
2. Human Performance– Organizational quality clock– Plant engineering personnel error rate
3. Rework– Unplanned change package revisions due to
design errors
8
What Is Measured Goal
Analysis Of Performance GreenWhy It Is Measured
How It Is Measured
Actions Taken Or Planned
Responsible Individual
Change Package QualityJuly 01, 2003
Jul-
02
Aug-0
2
Sep-0
2
Oct-
02
Nov-0
2
Dec-0
2
Jan
-03
Feb-03
Mar-03
Apr-03
May-
Jun
-03
1.50
1.75
2.00
2.25
DO
CU
ME
NT
AT
ION
QU
AL
ITY
Goal Lower Limit Goal Upper Limit Aggregate Average Lowest Performace area
Quality - Each change package is reviewed to determine how well the change package meets or exceeds expectations.
Change Package Quality is important to the safe, reliable and efficient operation of the power plant.
Change package quality has been improving.
Dan Dandreo - Supervisor Projects Design Engineering (Ext. 8382)
Green - 1 year weighted average between 1.75 and 2.00
Yellow - Trend value goes below 1.50
Red - Trend value goes above 2.00
Engineering Management continues to monitor change package quality as well as unplanned package revisions and incorporates improvements through the Engineering Support Training Program.
As each change package is reviewed by the responsible supervisor, a specially derived attribute is used to determine how well the change package meets or exceeds expectations. If the trend value falls below 1.75 the evaluator's expectations need to be reviewed. If the trend value goes above 2.00 the quality of the change packages needs to be improved.
All revisions to these change packages are evaluated against the same quality attributes.
9
3.0 Engineering Quality3.4 Engineering Technical Adequacy
Definition Analysis/Action
Goal CommentsPI Owner: D. C. LowePI Type: DataData Owner: M. A. FishData Source: EQRG DataDOE-ORP Owner: D. C. Bryson
This PI depicts evaluation of engineering document quality. This lagging performance indicator is used to track and trend the quality attributes required to establish a healthy Engineering Organization. The data is based on Engineering Quality Review Group (EQRG) assessment and subsequent rating of engineering documents reviewed using typical performance attributes.
Analysis: There is a recent decrease in quality apparent in the attention to detail in Engineering products reviewed over last 60 days. The 3 month rolling average continues to improve.
Action: Discipline leads will continue to target pre-release reviews for overall improvement and provide mentoring as required.
The goal is to produce all Engineering documentation such that if reviewed by the Engineering Quality Review Group the documentation would rate a <1.75.
Performance Attribute Weakness Current Month Average - July 2003
0
1
2
3
4
Rating
Engineering Technical Adequacy (August 2002 - July 2003)
1.00
2.00
3.00
4.00
Ratin
g
Overall Ave Rating 3.70 3.16 3.23 2.79 3.08 2.60 2.63 2.71 2.50 1.95 2.24 2.21
3-mo Rolling Ave 3.70 3.43 3.36 3.06 3.03 2.82 2.77 2.65 2.61 2.39 2.23 2.13
Goal 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75 1.75
Aug-02 Sep-02 Oct-02 Nov-02 Dec-02 Jan-03 Feb-03 Mar-03 Apr-03 May-03 Jun-03 Jul-03
Engineering Technical Adequacy
10
Month: JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DECPRS Section Resets 6 5 4 4 1 3PRS YTD Totals 6 11 15 19 20 23
PROJECT MANAGEMENT SECTION CLOCK
0
1
2
3
4
5
6
7
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
Nu
mb
er o
f S
ectio
n C
lock R
ese
ts
GOOD
YELLOW
DEFINITIONThe Project Management Section (PRS) defines events as any reset of the PRS Event Free Clock that is directly related to the operation of the facility. This does not include those administrative errors or training events related to PRS.
RANKING CRITERIA GREEN: No Section Clock Reset Events for the month. WHITE: < 2 Section Clock Reset Events for the month. YELLOW: < 4 Section Clock Reset Events for the month. RED: > 4 Section Clock Reset Events for the month.
OBJECTIVE/GOALThe goal is to maintain the highest standard of event mitigation by encompassing all of PRS performance duties into one category. PRS goal is to limit the number of events in this category to less than or equal to 12 events per year.
ANALYSIS CR Number Date Description CR 03-04831 6/04/03 Violation of Unescorted Access Requirements CR 03-04675 6/10/03 Violation of Procedure - Communication CR 03-05175 6/30/03 Violation of Procedure - Comply to Work Order
Analysis Responsibility - G. Wilson Achievement Responsibility - C. A. Hawley
Current Month:
Project Management Section Clock
11
Plant Engineering Personnel Error Rate
The Number of Significant Errors Recordedper 10,000 Hours Worked, 6 Mo.Rolling Average.
Notes:
Responsible Mgr: Data Provided By:PJ Inserra T Neidhold (C Leon)
Definition: Any event that resets our department clock.
Aug-02 Sep-02 Oct-02 Nov-02 Dec-02 Jan-03 Feb-03 Mar-03 Apr-03 May-03 Jun-03 Jul-03Number of Errors - - - 1 1 2 - 1 - - 1 - Hours Worked 8,235 7,989 9,757 7,006 7,371 8,884 8,257 9,771 9,005 12,371 11,239 8,126
0
0.2
0.4
0.6
0.8
1
1.2
Aug-01
Sep-01
Oct-
01
Nov-01
Dec-01
Jan-02
Feb-02
Mar-02
Apr-02
May-02
Jun-02
Jul-02
Aug-02
Sep-02
Oct-
02
Nov-02
Dec-02
Jan-03
Feb-03
Mar-03
Apr-03
May-03
Jun-03
Jul-03
Nu
mb
er o
f E
rro
rs p
er 1
0,0
00 H
ou
rs W
orked
There were no Human Performance Errors in the department for July.
12
What Is Measured Goal
Analysis Of Performance (Green)Why It Is Measured
How It Is Measured
Actions Taken Or Planned
Responsible Individual
Quality - Each change package revision is reviewed to determine the reason for the revision. The intent of this indicator is to track and measure how many change package revisions are required due to design errors.
Change Package Quality is important to the safe, reliable and efficient operation of the power plant.
Rolling 12 Month average
June 03: Green (2.7%)May 03: Green(3.2%)April 03: Green (3.4%)March 03: Green ( 3.6%)
Dan Dandreo - Supervisor Projects Design Engineering (Ext. 8382)
Green - Design errors average less than or equal to 5 %
Yellow - Design errors average greater than 5% and less than or equal to 10%.
Red - Design errors average above 10%
Engineering Management will monitor unplanned package revisions that are design errors and incorporate improvements through the Engineering Support Training Program. Since EIS will mandate a change in tracking mod revisions, evaluation and methodology may need revision.
Each change package revision is reviewed and graded by the responsible supervisor using the change package quality data base.
A 12 month running average is used to derive the above chart information.
Unplanned Change Package Revisions Due to Design Errors
0%1%2%3%4%5%6%7%8%9%
10%
Dec-0
2
Jan
-0
3
Feb
-0
3
Mar-0
3
Ap
r-0
3
May
-0
3
Ju
n-0
3
Ju
l-0
3
Au
g-0
3
Sep
-0
3
Oct-
03
No
v-0
3
Dec-0
3
12 month ave green lower limit= 5%
July 1, 2003
Unplanned Changed Package RevisionsDue to Design Errors
13
Path Forward
• Finalize “Safety System Performance” category
• Sub-group review of additional categories and selection of good practice examples
• Compile file report of good practice examples
• Target completion: March 1, 2004
15
Document QualityChange Package Quality
This PI tracks the quality of change packages as determined from a review performed by the responsible supervisor.
Strengths:• All the reviews and scoring are performed to a set of
predetermined criteria.• The PI includes descriptive information on what is
measured, how it is measured and what is considered good performance.
• An action level is indicated on the chart.• The change package customer rates the quality of
the product.
16
Suggested Improvement:• The PI would benefit from a clear visual indication that
shows the direction of increasing or decreasing performance.
• Total population of work packages reviewed vs. the percentage meeting quality expectations would provide information that could be used to benchmark among different sites.
• Information should be provided on how the one-year weighted average plot data is calculated. Major rating criteria should be stated.
Document QualityChange Package Quality
17
Document Quality Engineering Technical Adequacy
This PI tracks the quality of engineering documentation as determined by a quality review group comprised of senior level engineers.
Strengths:• All the reviews and scoring are performed to a set of
predetermined criteria.
• The PI includes a sub-chart that shows what attributes are reviewed and an average score for each attribute.
• A goal is indicated on the chart.
• A three-month rolling average is included so that trends are not masked by an individual month’s data.
18
Suggested Improvement:• The PI would benefit from a clear visual indication
that shows the direction of increasing or decreasing performance.
• Total population of documents reviewed vs. percentage meeting quality expectations would provide information that could be used to benchmark among different sites.
• Information should be provided on how the average document rating is calculated. Major rating criteria should be stated.
Document Quality Engineering Technical Adequacy
19
Human Performance Project Management Section Clock
This PI tracks the number of resets of the event free clock for a specific engineering group on a monthly basis.
Strengths:• The direction of increasing or decreasing quality is
indicated on the chart.
• The PI includes descriptive information on what is measured and a color rating scale.
• The performance goal is stated.
• Monthly score (color) is directly indicated on the charted data.
20
Suggested Improvements:• More definitive criteria should be included so that
objective performance can be compared to other sites. This should include a reporting threshold and examples for events that would be counted.
• A rolling average could be included so that trends are not masked by an individual month’s data.
• If the indicator included information on the number of hours worked in a given month, data could be compared to other sites using the error rate method of tracking human performance.
Human Performance Project Management Section Clock
21
Human Performance Plant Engineering Personnel Error Rate
This chart plots the rolling average personnel error rate per 10,000 hours worked for the specific engineering department.
Strengths:• The direction of increasing or decreasing quality is clearly
indicated by the color scale on the chart.
• The performance goal “green” is shown on the chart.
• Data are normalized (i.e., errors per 10,000 hours worked) so that information can be benchmarked among different sites.
• A rolling average is used so that trends are not masked by an individual month’s data.
22
Suggested Improvements:• The chart should include descriptive information on
what the threshold is for determining whether an error is significant. This was verbally provided as any entry into the plant corrective action system that had human error as a cause code and attributed to the Plant Engineering Department.
• The chart should include descriptive information on what is measured, why it is measured and how it is measured as will as recovery actions when performance goals are not met.
Human Performance Plant Engineering Personnel Error Rate
23
Rework Unplanned Change Package Revisions Due to Design Errors
This PI tracks the percentage of work packages that must be revised because design errors were detected during the review performed by the responsible supervisor.
Strengths:• The data are presented as a percentage vs. raw numbers
so that information can be benchmarked among different sites.
• The minimum performance goal is shown on the chart.
• A rolling average is used so that trends are not masked by an individual month’s data.
• The PI includes descriptive information on what is measured, how it is measured and what is considered good performance.
24
Rework Unplanned Change Package Revisions Due to Design Errors
Suggested Improvements: • The PI would benefit from a clear visual indication that
shows the direction of increasing or decreasing performance.
• Total population of work packages reviewed vs. the percentage meeting quality expectations would provide information that could be used to benchmark among different sites.
• Criteria should be provided for work package grading and the criteria for determining when a design error is significant, causing the package to be revised vs. inconsequential changes.
Recommended