Risk Assessment for QbD: Why FMEA Fails

If you tried a FMEA-based risk assessment for QbD, you will be familiar with the following story.

 Picture yourself and 4 scientists starting a meeting in a conference room at 4pm.

Scenario: Fighting over a rating of 3 versus 4

Project Manager: Let’s continue our Risk Assessment using FMEA. Today we’ll assess the detectability of Quality Attributes to generate Risk Priority Numbers. What do you think the detectability should be for this CQA (critical quality attribute)?

30 minutes later…

Cell culture Scientist: How about 3 for detectability because it’s better than 4?

Purification Scientist: I don’t know. It’s closer to 4 or maybe it’s 3.5.

Formulation Scientist: I still think it’s 4.

Quality:

 30 minutes later…

Project Manager: Thank you everyone. 3 CQA’s are tentatively done. We only have 30 more to go. I’ll set up another meeting tomorrow.

Everyone sighs…

 

3 months later: Most CQA’s are “critical.”

Project Manager:  We’ve finally finished our risk assessment. The result?

Risk Assessment

Quality Rep: (Panicking) Why are most CQA’s “critical?” You know that this translates to manufacturing and regulatory burden because we have to monitor and control these closely.

Scientists: That’s the manufacturing’s concern. We just need to meet our deadlines.

Silence…

Have you experienced this situation?  If you have done a Risk Assessment using FMEA, the chances are that you have. After spending 3 months in risk assessment meetings, too many attributes become critical. This defeats the purpose of conducting a risk assessment.

FMEA is not at fault. FMEA sessions become productive when participants have gained some experience in the product or process. However for development projects, detailed knowledge is usually not present.

Example of FMEA (Credit: ASQ)
Example of FMEA (Credit: ASQ)

Let’s take a step back. What is the main purpose of a risk assessment? To identify the top critical attributes and devise a control strategy around the high risk items. The main goal is not to miss the obvious risks.

What it is not —  list all things that could possibly go wrong. The chances are that the length of the list depends on our imagination and time.

Here are 3 main reasons why FMEA fails to be the optimal approach to conduct risk assessment for R&D teams.

1. Most attributes come out as “critical.

Risk calculation is biased towards occurrence. FMEA’s Risk calculation uses the equation below:

Risk Priority Number = Severity x Occurrence x Detectability( or Controllability)

Three factors in Risk:

1. Severity

2. Occurrence

3. Detectability (or Controllability)

 Let’s compare this with the classic definition of risk.

 Risk Index = Impact of Risk event x Probability of Occurrence

Two factors in Risk:

1. Impact

2. Probability

 Notice how FMEA’s risk priority number (RPN) has an extra factor — Detectability (or Controllability). According to the classic definition 1, definition 2, controllability or detectability are already included in the Occurrence factor.

For example, if the manufacturer has a good control strategy or detectability (i.e. with sensors) around drug potency, then the occurrence of  potency going out of specification will be low. The same holds true for the opposite condition. If the manufacturing process has low detectability or controllability for potency, then occurrence will be high. As you can see, detectability (or controllability) is a sub-factor of Occurrence. In other words, by adding detectability, FMEA’s formula inflates the Risk number. This is why much of the Quality Attributes end up in the red zone.

Recommendation: Stick with the classic risk definition: Impact x Occurrence.

 

2. At the development stage where scale-up details are not available, scientist do not yet understand the manufacturing process well enough to list realistic failure modes.

 Fighting over the ordinal rating of 3 versus 4 is meaningless at this stage. I’ve seen hours of disagreement over severity, occurrence, controllability and detectability. If you have an ordinal scale from 0 to 10, you will be wasting much time. Usually many organization just blindly adopt the AIAG standards from the automotive industry. At the least, risk assessment teams should develop their own ordinal scale so it applies. I’d like to point out that this is a big reason development scientists get frustrated with the risk assessment process.

Recommendation: Go with Low-Med-High or better yet, 0-1-3-9 scale (will discuss in the next article)

 

3. Mediocre Control Strategy

 After finishing FMEA, you will get a list of “high risk” or Critical Quality Attributes (CQA). But how can we control those CQA’s? Remember, the purpose of risk assessment is to devise a control strategy to prevent major failures. However, CQA’s are just symptoms or results. We can not directly control them. What we can directly control are CPP’s (Critical Process Parameters) and not CQA’s.

Control strategy or plan should focus on how to control the CPP’s or better yet, improve the process by finding the right design space. In order to do this, scientists must understand the relationship between CQA’s and CPP’s. If we mathematically model the manufacturing processes in a transfer function of Y=f(X),  CQA’s are Y’s (output) and CPP’s are X’s (input). This relationship is the design space of QbD.

With FMEA, control plan remains at a trivial level, such as equipment maintenance or monitoring plans. At the development stage, where the process itself still can be changed information on the Transfer Function, Y=f(X), is of more value — how process parameters affect the quality attributes. With this knowledge, scientists can design or order equipments around the appropriate range of these parameters.

 In summary, FMEA approach is a sub-optimal way to conduct a risk assessment for QbD.

Summary:

At the drug development stage, results of using FMEA for risk assessment are:

1. Most Quality Attributes will be “Critical” because FMEA’s risk formula inflates risk.

2. Scientists will waste time arguing over a meaningless evaluation scale.

3. Control strategy will be mediocre.

Most importantly, your organization will have missed out on the opportunity to understand the process.

FMEA sessions are productive when the team has some mature knowledge of the process. Some folks may argue the author doesn’t understand FMEA. During my PhD at Stanford University, I’ve researched, practiced, taught, and published peer-reviewed journal articles on FMEA. There is no doubt I highly appreciate the tool. However a tool is a tool — works great for the right application–may not for the wrong ones.

In a future post, I will share a better way of approaching Risk Assessment for Quality by Design.

UPDATE: The Process and Tool is here.

Subscribe to be the first one to learn about it.

Get New QbD Tips!
I agree to have my personal information transfered to AWeber ( more information )
Want More Tips from QbD Practitioners? Then join our newsletter!
We hate spam. Your email address will not be sold or shared with anyone else.
16 Comments