Thesis Defense Presentation

An Analysis of Risk Perception and the RPN Index within Failure Mode and

Effects Analysis

James L. Davie

Thesis Defense Presentation

18 August 2008

Motivation - 1

Risk management has become an important topic in business planning as global competition continues to grow.Manufacturing organizations concerned with risk management follow guidance information provided within their quality management system standards such as ISO 9001:2000, AS9100, ISO 13485:2003, TS16949.Guidance document ISO14971 exists as an aid to organizations in the implementation of a risk management program.One of the tools identified within the guidance information is the Failure Modes and Effects Analysis (FMEA), which establishes a methodology for the analysis of a specified system or subsystem.The FMEA is an engineering technique used to define, identify, and prevent potential failures from occurring within the system under analysis.

Motivation - 2

Within the FMEA tool the level of risk for a system component under analysis is assigned through the determination of the Risk Priority Number (RPN).The RPN is a composite score that is calculated as the product of three risk components; Severity, Occurrence, & Detection.These three risk components are assigned a value of 1-to-10 based on descriptions of each of the scale values. The scale values are based on generalized scales provided by the guidance information, or these scales can be modified to meet the unique requirements of a specific organization.The FMEA involves a systematic approach where a single element of a system is analyzed and all potential failure modes are investigated by the FMEA team.The team is comprised of representatives of all responsible departments associated with the system under analysis.

Research Objectives & Approach

This research focuses on the ability of individuals to evaluate risk associated with a given failure mode based their natural perception of risk and based on the rank established by the pre-defined component scales.

An experimental design was created to study these factors using established FMEA ranking scales to analyze the participants natural method of interpreting risk when presented in a narrative format vs. the FMEA method.

The FMEA is a team based approach where the output is a consensus of all team members, which can mask an individuals natural perception of risk.

For this reason, the experiment was conducted with individuals in order to gain a clear understanding of variation in the comparison between natural risk perception and FMEA risk analysis.

Discussion of Measurement Scales

Bowles (2003), Franceschini & Galetto (2007), and Puente et al (2001) identified that the FMEA uses a Likert scale, which is a type of ordinal scale that is commonly used in surveys, for the assessment of risk.

An ordinal scale measurement allows things to be assigned a value for the purpose of ordering a list of items from first to last. This scale does not indicate how much of a given characteristic an item may have, but simply that the item has more or less of the characteristic then another item within the list.

The magnitude of difference between two items cannot be interpreted from an ordinal scale.

Introduction to FMEA Rating Scales

The FMEA is a method used to determine the potential risk associated with a specified system or subsystem as determined by analyzing:

The Severity of potential injury to the customer.The Occurrence or likelihood the event will occur.The Detection or ability of the system to identify a failure.

These ratings are determined by using 1-to-10 Likert ranking scales that are given with descriptive examples to assist in the consistent evaluation of events.Teng & Ho (1996) state that these scales have little meaning to the team without a clear description of the impact associated with each level of associated risk.

Severity

According to Stamatis (2003) the severity is a rating of the seriousness of the effect of the potential system failure mode to the user and should be based on the worst case scenario.

Teng & Ho (1996) proposed a general severity scale guideline for organizations to follow when customizing the scale for their specific needs.

10 Catastrophic Effect (Death or Near Death)

9 & 8 Critical Effect (Immediate Medical Attention)

7 & 6 Major Effect (Medical Attention Required)

5 & 4 Minor Effect (First Aid Required)

3 & 2 Trivial Effect (Minor Physical Issue)

1 No Effect (No impact)

Occurrence

According to Stamatis (2003) the occurrence is a rating of the estimated number of failures that could occur associated with a specific failure mode.

The occurrence scale does not provide a narrative description of situations, but does provides a breakdown of failure rates in percentages, or even PPM.

Franceschini & Galleto (2007) claim that the FMEA method does not account for the production quantity. They state that in low level production situations that a single failure can cause the occurrence rating to increase dramatically. This claim is simply not true. The rating scales can be modified by any organization to reflect their individual needs. In the case of low volume manufacturing the scale should be redefined to meet the scope, and objectives of the organization.

Detection

According to Stamatis (2003) the detection rating describes the likelihood that the system will identify a failure before it reaches the customer.

If direct data is not available for a system, i.e. the system is under development, the data used for the purpose of risk analysis can be collected from existing similar process and/or technologies.

For example: A new gas tank is being designed for a motorcycle. No prototypes have been built, but based on the performance of existing gas tanks it is understood that pressure tested during production will be required as in the current design. This existing data can serve as a model for the risk analysis. If new pressure test capabilities become available this information can be use in the risk analysis as well.

Risk Priority Number (RPN)

The RPN is the product of the three risk component ordinal measurement scales: Severity, Occurrence, and Detection.

The RPN is used to rank order the identified potential failure modes within the defined system. This ranking is used to set a list of priorities for risk reduction and mitigation.

The RPN’s only meaning is to state that one failure mode has more or less risk than another failure mode. No other information can be inferred from this index.

Just as with the ordinal scales, no magnitude of change can be interpreted from one RPN value to another.

After risk reduction and mitigation activities have been completed the failure mode is reassessed and a new RPN is determined. No inference can be to the amount of risk that has been eliminated, only that the risk is less than the original analysis.

Problems with RPN Calculations

Puente et al (2001) outlines some of the major criticisms with the FMEA methodology.

The FMEA makes the assumption that only single point failures can occur within the system. This is why all of the guidance and instructional information for FMEA’s state that the tool should be used in conjunction with other methods such as a Fault Tree Analysis (FTA).There are many different combinations of S. O. & D. values that can generate the same RPN. Franceschini & Galletto (2001) points out that the FMEA assumes that the S. O. & D. scales should be equally weighted when determining the associated risk.There is no specified method for determining when action is required. Schmuland (2005) states that the “trip point” must be determined by the organization and that the justification and rationale for the selection has to be documented accordingly.

Problems with RPN Calculations

Bowles (2003) and Franceschini & Galetto (2001) identified that the RPN is a ordinal scale that does not completely cover the range of 1-to-1000.

A closer look at the RPN combination discovered that within the RPN range there exists only 120 individual outcomes. Within these outcomes some RPN values can be generated by as many as 24 different combinations of S. O. & D.

Ben-Daya & Raouf (1996) identify that the RPN does not satisfy the usual requirements of measure, and that there is no algebraic expression that reflects this scale.

The purpose of the RPN is to reflect the potential risk of a faulty product reaching the customer. However, can equivalent risk be accurately calculated 24 different ways?

Group Decision Making

The defined method of conducting a FMEA is to follow a team-based approach that includes representatives of all departments with responsibility for the system under analysis. The team format encourages discussion of the system from different view points and perspectives of the system.

Providing the opportunity for team members to openly discuss the potential failure modes of the system allows for all aspects to be considered when determining the severity, occurrence, and detection values.

Upon completion of the risk analysis the team is then required to determine what the appropriate threshold level should be for requiring risk reduction actions.

Risk Based Decision Making

Wong (2005) identified that predictors of risk-taking behaviors do not exert direct effects on risky decision-making. Rather, their effects are mitigated by risk perception and risk propensity.

• Risk perception is the decision maker’s assessment of inherent risk in a situation, i.e. natural perception.

• Risk propensity is the decision maker’s tendency to take or avoid risk.

MacCrimmon & Wehrung (1990) show that more mature decision maker’s in terms of age and seniority, have a lower risk propensity and are more risk averse then those subjects that were considered to be less mature.

Methodology

A sampling of employee’s from manufacturing companies were used in order to compare the natural human perception of risk when presented in a narrative situation vs. the perception when presented in the FMEA format.

The results of this study will assist in establishing a relationship between the way people naturally perceive risk and the way risk is analyzed in the FMEA matrix.

This comparison will help to determine if a correlation exists between innate risk perception and FMEA risk analysis.

Methodology

FactorsThe comparison of the “Narrative Survey” vs. “FMEA Matrix” was the first factor of this experiment. The second factor is the order in which the questions were presented.The same questions were used for the narrative survey and the FMEA matrix. The order of the question were randomized through the creation of six different ordered versions to generate twenty-five unique order pairs.

CovariatesCovariates within designed experiments are uncontrolled variables that influence the response. The covariates for this research were gender, educational background, and FMEA experience. These items were analyzed through an ANOVA analysis.

Participants

The participants for this study included employees of manufacturing organizations representing different areas of system responsibility.

Participation in this study was completely voluntary and the participants had the right to withdraw from the study at any time.

It was clearly communicated that participating or not participating in the study would not impact their employment status.

Each participant was presented with a consent document that detailed their rights as a participant, and to obtain approval for the response information to be used as part of this study.

Participant Training

Introductory level training was provided to the participants on the FMEA method describing the concepts of Severity, Occurrence, and Detection.

A generalized scale was provided for use by the participants with the surveys to assist in the selection of the rank values.

The survey consisted of unbiased and non-leading closed form questions designed to study the difference in the perceived risk associated with a situational description.

The participants were provided a ten-minute break between the narrative and the FMEA matrix sections of the survey.

The ordering of the questions was randomized from the narrative portion to the FMEA matrix to remove any bias that could be introduced from the repeated ordering of the questions.

Experimental Procedure

Participants were asked to complete the following:Read and sign the provided consent form indicating that they understand their rights as a participant.Complete the three demographic questions at the top of the survey.Complete the twenty-seven question narrative survey to assess your perception of risk associated with the situational circumstances.Take a ten-minute break.Read the training information provided for the FMEA form.Complete the twenty-seven question FMEA matrix to assess your perception of risk associated with the situational circumstances.Each of the questionnaires should take approximately 15-30 minutes, but you may take longer if needed.

Results – Demographic Measure Analysis

A group of 100 candidates were identified as potential participants for this study. Of these, 34 people provided responses in a timely manner. Each of these participants completed all three of the demographic questions.

Gender

Of the 34 respondents twenty-five were 25 Male, 73.5%, and 9 were Female, 26.5%

The gender demographic was the only demographic found through the ANOVA analysis to impact the perception of Severity and Occurrence. However, gender did not impact the perception of detection.

Results – Demographic Measure Analysis

EducationBased on the collected data the Mean, Median, and Mode indicate that the average participant has completed a Bachelor’s degree. The educational level of the participants was not found to have an impact on the perception of Severity, Occurrence, or Detection.

Years of ExperienceBased on the collected data the participants Mean experience level was found to be 9.7years. However, the Median was 7 years and the Mode was 0.5 years indicated that the this demographic is not normally distributed. The years of experience was not found to have an impact on the perception of Severity, Occurrence, or Detection.

Analysis of Risk Perception ResponsesNo Experience

Participants with no prior experience in FMEA were not able to accurately assign risk to the narrative survey, or to the FMEA survey with any correlation to the planned RPN values.

This indicates the importance of clear and effective training of personnel responsible for conducting risk analysis activities.

RPN

R_RPN

10008006004002000

1000

800

600

400

200

0

S 0.478224R-Sq 48.4%R-Sq(adj) 48.3%

Experimental RPN Condition vs RPN Response when Exp = 0logten(R_RPN) = - 0.01938 + 0.9155 logten(RPN)

RPN

Sco

re

10008006004002000

120

100

80

60

40

20

0

S 0.375057R-Sq 37.1%R-Sq(adj) 37.0%

Experimental RPN Condition vs SCORE Response when Exp = 0logten(Score) = 0.3143 + 0.5690 logten(RPN)

Analysis of Risk Perception ResponsesTwo Years Experience

Here the participants were presented with the same problem, but generated a very different result.

When people are trained they are able to assess risk as planned by the FMEA.

However, this assessment of risk does not correlate with their actual perception of risk as shown in the RPN condition vs. Score Response chart.

RPN

R_RPN

10008006004002000

900

800

700

600

500

400

300

200

100

0

S 0.129724R-Sq 95.3%R-Sq(adj) 95.1%


RPN

Sco

re

10008006004002000

140

120

100

80

60

40

20

0

S 0.484197R-Sq 49.4%R-Sq(adj) 47.4%

Experimental RPN Condition vs SCORE Response when Exp = 2logten(Score) = - 0.6312 + 0.9114 logten(RPN)

Analysis of Risk Perception ResponsesEight Years Experience

With this group the concern is that the RPN interpreted low risk, and the FMEA characterized the situation as low risk.

Therefore, the failure mode will not be considered for additional risk analysis and reduction.

However, when asked to rate the risk without the FMEA these situations were considered to be a high risk.

RPN

R_RPN

10008006004002000

1000

800

600

400

200

0

S 0.189741R-Sq 90.8%R-Sq(adj) 90.6%


RPN

Sco

re

10008006004002000

90

80

70

60

50

40

30

20

10

0

S 0.398748R-Sq 22.0%R-Sq(adj) 20.5%

Experimental RPN Condition vs SCORE Response when Exp = 8logten(Score) = 0.5259 + 0.4116 logten(RPN)

Analysis of Risk Perception ResponsesTwenty-Five Years Experience

This data set shows the strongest correlation of both the RPN response and the Score response to the planned condition.

Here, the concern is that the RPN was low and the FMEA characterized the failure modes as low risk. Therefore no additional action is required.

However, when asked to rate the risk without the FMEA some situations were perceived as higher risk than the RPN indicates.

RPN

R_RPN

10008006004002000

1200

1000

800

600

400

200

0

S 0.0135864R-Sq 100.0%R-Sq(adj) 100.0%


RPN

Sco

re

10008006004002000

200

150

100

50

0

S 0.363871R-Sq 62.5%R-Sq(adj) 61.0%

Experimental RPN Condition vs SCORE Response when Exp = 25logten(Score) = - 0.3668 + 0.8951 logten(RPN)

Discussion of Results

The results of this research indicates that an individual's perception of risk is influenced differently by each of the three components utilized by the FMEA model.

In the application of the model, risk analysis is completed in a group setting where all aspects of the system can be discussed and a consensus decision of risk can be made.

Using the team based approach may cause the resultant decisions to be normalized with respect to risk perception, personal/experience, and cultural bias.

Based on the results of participants with no experience it is important to pair new team member with strong team leaders and/or team facilitators to provide training and offer mentoring through the process.

Discussion of Results

Overall, these experimental findings confirm that the RPN does not accurately reflect a person’s innate perception of risk.Significant findings show that the participants of the survey assessed risk in a dissimilar fashion when the same situation was presented within the context of the FMEA structure.The use of the FMEA form allows people to visualize the components of the situation and make informed decisions of the associated risk based on the severity, occurrence, and detection.The FMEA form that was included in the survey did not follow the exact format of the FMEA. For the purpose of this investigation the failure mode, effect, and detection methods were organized in a one-to-one ratio.The concern is that low planned RPN values that would not require further action were perceived as high risk when assessed without the FMEA.

Future Study – Action Point Method

An action point method will need to be established as a bivariate or univariate analysis of Severity, Occurrence, Detection where the Severity and Occurrence interaction can be incorporated into the calculation.

An alternate action point method could utilize a calculation were only Severity and Occurrence determine if risk reduction activities are required.

Future Study

An interesting factor to study with a group dynamic experiment would be the management of strong personalities and how they influence the decision making process.

Another interesting aspect to the group dynamic experiment would be the results of the FMEA analysis when conducted with and without a team facilitator.

It would be of significant interest to study the way that severity impacts occurrence through the group dynamic when compared to the individuals results of this study.

Future Study

In 2005, Kin Fai Ellick Wong published the results of a psychological study, which found that the differences between eastern and western cultures significantly impact the way a person perceives risk.

Based on the findings of Wong (2005) executing this study with eastern participants would provide important information regarding the consistency of the FMEA tool across the globe.

As more companies continue to expand their manufacturing capabilities across the global market, risk analysis activities that are conducted at international facilities by local personnel can become a significant challenge to manage.

In this situation variations in risk perception, and the risk reduction activities could become inconsistent from one facility to another.

Closing

Thank you for your time.

I would like to ask if anyone has questions or comments regarding risk analysis / risk perception.

Acknowledgement

Dr. Li Lin, Ph.D

for acting as my advisor for this research.

Dr. Harrison Kelly, Ph.D

for his continued guidance and advise throughout the research development and analysis.

Technology

Thesis Defense Presentation