Upload
noel-nolan-benney
View
217
Download
1
Tags:
Embed Size (px)
Citation preview
Company Confidential 1
Registration Management Committee (RMC)
2010 Report on Oasis Data 2007-2010
Long Beach, CASeptember 13 -14, 2011
R. Darrell TaylorRaytheon
Long Beach, CA September 13-14, 2011
Registration Management Committee (RMC)
2
Executive Summary
• Baselines have been established for 2007 – 2010
• No Feedback data, changing metric
• Review Proposed revision to the CB Report Card
Registration Management Committee (RMC)
3
Agenda• OASIS Entries – Audits
• OASIS Nonconformities
• AS9100 Elements
• Feedback
• Monitoring Model
Company Confidential 4
Registration Management Committee (RMC)
Entries – Audits
Long Beach, CA September 13-14, 2011
Registration Management Committee (RMC)
5
Comparison of Entries 2007 -2010
• Entries:
– 2007 – 5114
– 2008 – 5972 ^16%
– 2009 – 6969 ^17%
– 2010 – 7502 ^7.6%
– -2011 – 32538/1/2011
2007 2008 2009 2010
0
1000
2000
3000
4000
5000
6000
7000
8000
5114
5972
6969
7502
Total Entries
Increase slowing down
Registration Management Committee (RMC)
6
Comparison of Types2007 -2010
SUR REC INT
2007 3364 888 862
2008 3977 923 1072
2009 4421 1486 1062
2010 5131 1506 865
SUR=Surveillance
REC = Recertification
INT = Initial
2011 – 72%, 21 %, 7%
SUR REC INT2007 0.66 0.17 0.172008 0.67 0.15 0.182009 0.63 0.21 0.152010 0.68 0.20 0.12
Registration Management Committee (RMC)
7
Standards Entered Into Database
2007 2008 2009 20100.0000.0050.0100.0150.0200.0250.0300.0350.0400.0450.050 AS9120
AS9110NBR15100:2004EN9100:2003/S1EN9120:P2/S1EN9100:P2/S1AS9100C
AS9100B:
2007- 95%2008 - 95%2009 - 94%2010 -92%
38 AS9100C – 2010; 225 in 2011 as 05 8/1/2011
Registration Management Committee (RMC)
8
CRB Comments - 2010• 2007 – 2010 Data baselined
Going to Rolling Format
Company Confidential 9
Registration Management Committee (RMC)
Nonconformities
Long Beach, CA September 13-14, 2011
Registration Management Committee (RMC)
10
Pareto of Total Nonconformities
Totals
2010 – 20,009
2009 -20,980
2008 -18,803
2007 – 14,957
2011 – 65% 2 or less
Registration Management Committee (RMC)
11
OASIS Ratio of Total Findings
Violates Normal DistributionStrongest fit 3 Parameter Lognormal
Identify CB’s in the bottom 5% writing nonconformities
157.5135.0112.590.067.545.022.50.0
Median
Mean
2.82.62.42.22.0
1st Quartile 0.000Median 2.0003rd Quartile 4.000Maximum 157.000
2.594 2.765
2.000 2.000
3.701 3.822
A-Squared 498.36P-Value < 0.005
Mean 2.680StDev 3.761Variance 14.143Skewness 11.681Kurtosis 399.522N 7467
Minimum 0.000
Anderson-Darling Normality Test
95% Confidence Interval for Mean
95% Confidence Interval for Median
95% Confidence Interval for StDev95% Confidence I ntervals
Summary for TOTAL
1751501251007550250
99.99
99
95
80
50
20
5
1
0.01
TOTAL
Pe
rce
nt
Mean 2.680StDev 3.761N 7467AD 498.357P-Value <0.005
Probability Plot of TOTALNormal - 95% CI
Registration Management Committee (RMC)
12
Total - Comments
• Field “Family of Standards” states ISO9001:2000 s/b 2008?
• 35 (<.5%)Entries with N/A in Numbers, what does that mean?– 2 Initials, 1 surveillance, 32 recertification
Registration Management Committee (RMC)
13
Ratio Majors/Entries
On average, one Major is written every fifth audit – NOTE: Average down to .18 per audit for 2010 (1/5 audits)
No Change in 2011 (.19) .22 for AS9100C audits
Count 6738 431 298Percent 90.2 5.8 4.0Cum % 90.2 96.0 100.0
RESP_TOTAL_MA Other10
8000
7000
6000
5000
4000
3000
2000
1000
0
100
80
60
40
20
0
Count
Perc
ent
Pareto Chart of RESP_TOTAL_MA
Totals
2010 – 1387
2009 -1135
2008 – 976
2007 - 870
Registration Management Committee (RMC)
14
OASIS Ratio of Majors
NOT Normal Distribution strongest 3 parameter loglogistic
Identify CB’s in the bottom 5% writing majors
1086420
Median
Mean
0.200.150.100.050.00
1st Quartile 0.0000Median 0.00003rd Quartile 0.0000Maximum 11.0000
0.1689 0.2026
0.0000 0.0000
0.7309 0.7547
A-Squared 2197.91P-Value < 0.005
Mean 0.1858StDev 0.7426Variance 0.5515Skewness 6.2414Kurtosis 50.7385N 7467
Minimum 0.0000
Anderson-Darling Normality Test
95% Confidence Interval for Mean
95% Confidence Interval for Median
95% Confidence Interval for StDev95% Confidence I ntervals
Summary for RESP_TOTAL_MA
12.510.07.55.02.50.0-2.5-5.0
99.99
99
95
80
50
20
5
1
0.01
RESP_TOTAL_MAP
erc
en
t
Mean 0.1858StDev 0.7426N 7467AD 2197.911P-Value <0.005
Probability Plot of RESP_TOTAL_MANormal - 95% CI
Registration Management Committee (RMC)
15
Ratio Minors/Entries
On average, 2.5 Minors are written every audit2011 – down to 2.12
AS9100C – 3.18
Count 149 2932191 1307 1190 906 553 399 263 217Percent 2.0 3.929.3 17.5 15.9 12.1 7.4 5.3 3.5 2.9Cum % 96.1 100.029.3 46.8 62.8 74.9 82.3 87.7 91.2 94.1
RESP_TOTAL_MI Other876543210
8000
7000
6000
5000
4000
3000
2000
1000
0
100
80
60
40
20
0
Count
Perc
ent
Pareto Chart of RESP_TOTAL_MI
Totals
2010 – 18,627
2009 -19,414
2008 -17,393
2007 –14,087
Registration Management Committee (RMC)
16
OASIS Ratio of Minors
NOT Normal DistributionLognormal Distribution
Identify CB’s in the bottom 5% writing minors
135.0112.590.067.545.022.50.0
Median
Mean
2.62.52.42.32.22.12.0
1st Quartile 0.000
Median 2.0003rd Quartile 4.000Maximum 151.000
2.415 2.573
2.000 2.000
3.424 3.535
A-Squared 481.38
P-Value < 0.005
Mean 2.494StDev 3.479Variance 12.102
Skewness 12.705Kurtosis 464.332N 7468
Minimum 0.000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence I nterval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for RESP_TOTAL_MI
1501251007550250
99.99
99
95
80
50
20
5
1
0.01
RESP_TOTAL_MIP
erc
en
t
Mean 2.494StDev 3.479N 7468AD 481.379P-Value <0.005
Probability Plot of RESP_TOTAL_MINormal - 95% CI
Registration Management Committee (RMC)
17
Evaluation of Audit Type INT, REC, SUR
• Chi Squared Test returned a P-Value = 0.069
Major AVG SD Count MIN MED MAX
INT .2514 .7991 217 0 0 6
REC .2609 .9707 384 0 0 11
SUR .1532 .6492 786 0 0 10
Minor
INT 3.907 4.023 3372 0 3 29
REC 3.369 3.980 4966 0 2 66
SUR 2.005 3.091 10284 0 1 151
Registration Management Committee (RMC)
18
AS9100 Nonconformities Top 10
Top Eight 2010 same as 2009 & 2008
NO CHANGE
Registration Management Committee (RMC)
19
Feedback Analysis
2010 average 103 per month2009 average 112 per month2008 average 110 per month
2007 2008 2009 20100
500
1000
1500
2000
2500
As of June 6% of 2010 Feedback Responses still open
ON HOLD
Registration Management Committee (RMC)
20
Certification Body Statistical Assessment Model
ANAB Audits - 6 Criteria
OASIS Results - 3 Criteria
Feedback - 3 Criteria
General - 1 Criteria
ANAB Performance -4
ON HOLD
Registration Management Committee (RMC)
21
Threshold Violations
Violations0
2
4
6
8
10
12
14
16 • RMC & ANAB need to research
– >6
• ANAB to report to RMC
– 4-5
• ANAB to investigate
– 2-3
• No Action
– <2
Company Confidential
Registration Management Committee (RMC)
Oversight Results2011 Major & Minor Numbers2 WA – 1 Minor / 0 Majors & 2 Minors / 0 Majors2 OA – 3 Minors / 0 Majors & 10 Minors / 3 Majors
2009 Major & Minor Numbers 2 WA –1 Minor /4 Majors 2 OA –7 Minors /0 Major
NCR trends – Not verifying OASIS admin. on site, OASIS accuracy, depth of audit, not documenting all NCRs, various checklist issues
CB Activity & AEA’sStandard Sites AEA’s9100 100 149110 Applicant 09120 8 8
OASIS Data Supplemental ActivityAB Actions: 2010 – March WA was a 2009 re-auditCB has received 2 OA’s per year for two years.1 major NCR written outside of an assessment
RMC Member: (Supplemental oversight data)
2 events, 4 minor NCR’s
CB Name: XXX -2011
AUDITS 74TOT MAJORS 75MAJOR AVERAGE 1.01TOT MINORS 275MINOR AVERAGE 3.72FEEDBACK 22AVERAGE DAYS OPEN 9.31NUMBER STILL OPEN 0
Company Confidential
Registration Management Committee (RMC)
CB Name: XXX -2011
Certification Body Performance ReviewCertification Body: XXXXXXXXXXXX
METRICS RULEICOP AUDIT RESULTS 2008 2009 2010 2011Number of Audits ### ### ### ###Minors >1SDMajors >1SDRatio of Minors to Audits >1SDRatio of Majors to Audits >1SDRatio of Total to Audits >1SDMinors issued outside audit >1Majors issued outside audit >0Increased audits >2
PERFOMANCE METRICSNegative Shift in Audit Totals >-1SDEscalated N/Cs >0Customer Complaints >0Suspensions (2 years) >0Negative Shift in Violations > - 1SDGENERAL (RISK) OPEN
VIOLATIONS
OASIS METRICSMinors ### ### ### ###Majors ### ### ### ###Total ### ### ### ###Ratio of Minors to Audits <1SDRatio of Majors to Audits <1SDRatio of Total to Audits <1SD
FEEDBACK MANAGEMENTClosed Average Closure >30 OPEN Average open days >30OPEN > 1 Year >365
ANAB Additional Comments
Registration Management Committee (RMC)
24
Questions