Inconvenient Truth: RPN in QbD Risk Assessment

Posted On

You heard the story about the troubled QbD Risk Assessment meeting.

 Picture yourself and 4 scientists starting a meeting in a conference room at 4pm.

Scenario: Fighting over a rating of 3 versus 4

Project Lead: Let’s continue our Risk Assessment using FMEA. Today we’ll assess the severity of Quality Attributes to generate Risk Priority Numbers. What do you think the severity should be for this CQA (critical quality attribute)?

 

30 minutes later…

Upstream Scientist: How about 3 for severity because it’s better than 4?

Downstream Scientist: I don’t know. It’s closer to 4 or maybe it’s 3.5.

Formulation Scientist: I still think it’s 4.

Quality Rep: …(silence)

 

30 minutes later…

Project Lead: Thank you everyone. 3 CQA’s are tentatively done. We only have 30 more to assess. I’ll set up another meeting tomorrow.

Everyone sighs…”

 During the meeting, you find asking yourself,

“How can we finish a QbD Risk Assessment meeting without getting stuck in a rating argument?”

 

We’ll address this question today.

In this article, we’ll cover:

• What is RPN?
• The mathematical flaw behind FMEA’s RPN calculation
• How you calculate RPN may change your Control Strategy
• Which RPN scale you should use for QbD Risk Assessments
• What is the main goal of a QbD Risk Assessment?

 

First, let’s cover the definitions.

 

What is RPN (Risk Priority Number)?

 

For those who are new to Risk Assessments or FMEA (Failure Mode and Effects Analysis), RPN is an acronym for Risk Priority Number.

In the context of QbD Risk Assessment, RPN is basically a score that tells you how “risky” a process parameter or a quality attribute is. The higher the number, the higher the “risk” is.

 

There are 2 popular ways to calculate a RPN — multiplying 2 or 3 variables. 

• For Traditional FMEA: RPN = Severity x Occurrence x (Detectability or Controllability)
• Recommended for QbD Risk Assessment: RPN = Severity x Occurrence

 So far so good. High RPN score means High Risk.

 

Here’s where RPN gets interesting:

 

RPN is a simple multiplication of 2~3 numbers? Ok, but how do we come up with those numbers?

Let me explain further: How do I put a number to Severity? Some folks use a scale of 1 to 10, others, 1 to 3. This is the same with Occurrence.

From my survey, I found 3 popular practices: All use ordinal or a Likert rating scales

1. 3-level Rating: 1 (Low) – 2 (Med) – 3 (High) or 1-3-9
2. 10-level Rating: 1-2-3-4-5-6-7-8-9-10
3. Hybrid (i.e. 1-3-6-9)

 Here come the Inconvenient Truths: Using Ordinal Scales to calculate RPN’s

 

1. RPN does not preserve rank order. Multiplying different rankings into a single number (RPN) is misleading.

 

Donald Wheeler calls this practice, “utter and complete non­sense.” In Problems with Risk Priority Numbers,Quality Digest, 2011, he explains why:

 

“The lack of a distance function and the lack of an absolute zero will combine to result in inconsistencies where both serious and trivial problems have the same RPN value (muddying up results), and where some trivial problems end up with larger RPN values than other, more serious, problems (mixed-up ranking, aka inflated or deflated scores).”

 

For example, you can’t reliably calculate the safety rating of a driving situation by multiplying the type of a car (minivan-sedan-convertible) and the type of driver (adult-senior-teenager).

Can we reliably assess which situation is more dangerous: Sedan x Teenager versus Convertible x Senior? Not really.

So Wheeler proposes that we should look at the Severity and Occurrence separately – as separate dimensions.

I was happy to learn that Lean QbD Risk Assessment enables me to do this in the Control Strategy tab.

 

 

This makes a lot of sense. Because even if RPN’s are the same, the combination of impact and occurrence could be very different. For example, if RPN is 30 there are cases where Impact=3, Occurrence=10 or another case where Impact=10, Occurrence=3.

Should we have the same control strategy for both of these cases? No, since the rationale behind these cases are different.

It’s best for us development or process scientists to look at severity and occurrence separately.

 

2. You do not a need 10-point scale at the Development Stage.

 

I disagree with Wheeler on one point. FMEA is imperfect but if the scale has less resolution, it still serves its purpose as a tool to distill high-risk items.

Many folks forget this important point. The goal of a FMEA  is to ensure we find the high-risk modes – not making the longest list of “possible” mishaps.

Let’s go back to the Vehicle Safety case. We may quibble about the case of: Sedan x Teenager vs. Convertible x Senior. But we can agree that Convertible x Teenager is riskier than Minivan x Adult. So we focus on preventing Convertible x Teenager combination – the high risk scenario.

So what’s the implication for us?  Focus of QbD Risk Assessment should be on de-risking the high-risk quality attributes and process parameters.

Ok then will our risk results be different depending on the type of scales we use?

 

3. Are RPN’s affected by the number of scale points used (Low-Med-High VS 1-10)?

 

John Dawes asked a similar question. Dawes (2008) published the study, “Do Data Characteristics Change According to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales.”

What was the study’s conclusion? The ranking results didn’t change much despite using varying levels of point scales.

This is good news for us doing QbD Risk Assessment at the development stage – where not much information is available yet. Use the appropriate resolution at the development stage. Don’t overcomplicate QbD risk assessment by using a 10-point scale. Use a 3-scale rating for QbD Risk Assessement at the development stage.

 

Conclusion: 

You need a RPN scale commensurate at the level of information available. If you are in development stage, High-Med-Low is the best. Separate severity and occurrence (and detectability if you use it) rankings. Dr. Wheeler wholeheartedly agrees with this conclusion.

“When working at the design phase…  you may use the three scales, and use the rankings, but you should not use the RPN values.”  D. Wheeler (2011)

 Let’s focus on the main goal of the Risk Assessment in Quality by Design. At the development stage, our goal should be on linking QTPP-CQA-CPP/CMA together. The Lean QbD tool guides you in doing exactly this.

 

In the next post, I’ll share how others are applying this in their practices.

 

References:

Gilchrist, Modelling Failure Modes and Effects Analysis, International Journal of Quality and Reliability Management 10 (5), 16-23, 1993.

Kmenta, Scenario-based FMEA Using Expected Cost, A new perspective on evaluating risk in FMEA, IIE Workshop,January 22, 2002.

Kmenta and K. Ishill, Scenario-Based Failure Modes and Effects Analysis using Expected Costs, Journal of Mechanical Design 126, 1027-1036, 2004.

Dawes, Do Data Characteristics Change According to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales,” International Journal of Market Research, 2008