Risk Assessment for QbD: Why FMEA Fails
If you tried a FMEA-based risk assessment for QbD, you will be familiar with the following story.
Picture yourself and 4 scientists starting a meeting in a conference room at 4pm.
Scenario: Fighting over a rating of 3 versus 4
Project Manager: Let’s continue our Risk Assessment using FMEA. Today we’ll assess the detectability of Quality Attributes to generate Risk Priority Numbers. What do you think the detectability should be for this CQA (critical quality attribute)?
30 minutes later…
Cell culture Scientist: How about 3 for detectability because it’s better than 4?
Purification Scientist: I don’t know. It’s closer to 4 or maybe it’s 3.5.
Formulation Scientist: I still think it’s 4.
Quality:
30 minutes later…
Project Manager: Thank you everyone. 3 CQA’s are tentatively done. We only have 30 more to go. I’ll set up another meeting tomorrow.
Everyone sighs…
3 months later: Most CQA’s are “critical.”
Project Manager: We’ve finally finished our risk assessment. The result?
Quality Rep: (Panicking) Why are most CQA’s “critical?” You know that this translates to manufacturing and regulatory burden because we have to monitor and control these closely.
Scientists: That’s the manufacturing’s concern. We just need to meet our deadlines.
Silence…
Have you experienced this situation? If you have done a Risk Assessment using FMEA, the chances are that you have. After spending 3 months in risk assessment meetings, too many attributes become critical. This defeats the purpose of conducting a risk assessment.
FMEA is not at fault. FMEA sessions become productive when participants have gained some experience in the product or process. However for development projects, detailed knowledge is usually not present.
Let’s take a step back. What is the main purpose of a risk assessment? To identify the top critical attributes and devise a control strategy around the high risk items. The main goal is not to miss the obvious risks.
What it is not — list all things that could possibly go wrong. The chances are that the length of the list depends on our imagination and time.
Here are 3 main reasons why FMEA fails to be the optimal approach to conduct risk assessment for R&D teams.
1. Most attributes come out as “critical.
Risk calculation is biased towards occurrence. FMEA’s Risk calculation uses the equation below:
Risk Priority Number = Severity x Occurrence x Detectability( or Controllability)
Three factors in Risk:
1. Severity
2. Occurrence
3. Detectability (or Controllability)
Let’s compare this with the classic definition of risk.
Risk Index = Impact of Risk event x Probability of Occurrence
Two factors in Risk:
1. Impact
2. Probability
Notice how FMEA’s risk priority number (RPN) has an extra factor — Detectability (or Controllability). According to the classic definition 1, definition 2, controllability or detectability are already included in the Occurrence factor.
For example, if the manufacturer has a good control strategy or detectability (i.e. with sensors) around drug potency, then the occurrence of potency going out of specification will be low. The same holds true for the opposite condition. If the manufacturing process has low detectability or controllability for potency, then occurrence will be high. As you can see, detectability (or controllability) is a sub-factor of Occurrence. In other words, by adding detectability, FMEA’s formula inflates the Risk number. This is why much of the Quality Attributes end up in the red zone.
Recommendation: Stick with the classic risk definition: Impact x Occurrence.
2. At the development stage where scale-up details are not available, scientist do not yet understand the manufacturing process well enough to list realistic failure modes.
Fighting over the ordinal rating of 3 versus 4 is meaningless at this stage. I’ve seen hours of disagreement over severity, occurrence, controllability and detectability. If you have an ordinal scale from 0 to 10, you will be wasting much time. Usually many organization just blindly adopt the AIAG standards from the automotive industry. At the least, risk assessment teams should develop their own ordinal scale so it applies. I’d like to point out that this is a big reason development scientists get frustrated with the risk assessment process.
Recommendation: Go with Low-Med-High or better yet, 0-1-3-9 scale (will discuss in the next article)
3. Mediocre Control Strategy
After finishing FMEA, you will get a list of “high risk” or Critical Quality Attributes (CQA). But how can we control those CQA’s? Remember, the purpose of risk assessment is to devise a control strategy to prevent major failures. However, CQA’s are just symptoms or results. We can not directly control them. What we can directly control are CPP’s (Critical Process Parameters) and not CQA’s.
Control strategy or plan should focus on how to control the CPP’s or better yet, improve the process by finding the right design space. In order to do this, scientists must understand the relationship between CQA’s and CPP’s. If we mathematically model the manufacturing processes in a transfer function of Y=f(X), CQA’s are Y’s (output) and CPP’s are X’s (input). This relationship is the design space of QbD.
With FMEA, control plan remains at a trivial level, such as equipment maintenance or monitoring plans. At the development stage, where the process itself still can be changed information on the Transfer Function, Y=f(X), is of more value — how process parameters affect the quality attributes. With this knowledge, scientists can design or order equipments around the appropriate range of these parameters.
In summary, FMEA approach is a sub-optimal way to conduct a risk assessment for QbD.
Summary:
At the drug development stage, results of using FMEA for risk assessment are:
1. Most Quality Attributes will be “Critical” because FMEA’s risk formula inflates risk.
2. Scientists will waste time arguing over a meaningless evaluation scale.
3. Control strategy will be mediocre.
Most importantly, your organization will have missed out on the opportunity to understand the process.
FMEA sessions are productive when the team has some mature knowledge of the process. Some folks may argue the author doesn’t understand FMEA. During my PhD at Stanford University, I’ve researched, practiced, taught, and published peer-reviewed journal articles on FMEA. There is no doubt I highly appreciate the tool. However a tool is a tool — works great for the right application–may not for the wrong ones.
In a future post, I will share a better way of approaching Risk Assessment for Quality by Design.
UPDATE: The Process and Tool is here.
Subscribe to be the first one to learn about it.
Sun,
Do you have any good references for teaching the principles of FMEA?
Alan
Alan, After numerous FMEA sessions, I have a different take on the tool which I will share in the future.
Traditional usage/template is here: http://asq.org/learn-about-quality/process-analysis-tools/overview/fmea.html
Again, the approach is different for new development projects.
Sun Kim
Manager, Master Black Belt in Quality-by-Design, Agile Lean Six Sigma and Design Thinking
Top Contributor
If you have a unique approach, please share! I will share my approach soon.
Delete 12 days ago
Sarah Betterman
Sarah
Sarah Betterman
Experienced Formulation and Process Development Scientist
I agree with your perspective, Sun. The formality of risk assessment should align with the level of product and process understanding. FMEA definitely has its place, but later in the development process. To simplify even further, I have used a two-level Low-High ranking system. However, I look forward to hearing the details of your approach as I am always looking for ways to improve!
Like (1) Reply privately Flag as inappropriate 12 days ago Line Lundsberg-Nielsen likes this
Stefano Selva
Stefano
Stefano Selva
R&D Pharma Formulation Development at Angelini
Great discussion! It is like a loop I always have to dealt with, everytime! The importance of the FMEA approach is surely not under discussion. Despite of this, in my opinion, relying just on the FMEA approach in the early-medium stages of development (prior of scale up) could bring to serious risk (it is like a ‘bias’ related to the scientists’ prior knowledge). Using an approch consisting on data mining / ‘passive’ multivariate analysis of ‘previous and similar formulation and processes’ (a simple PCA for example?) could be, in my opinion, a good way to asses CPPs and Critical parameters of raw materials (and bulk variability!). This analysis (based on real/process data and products) could help defining the most significant factors to study on a DoE for new developments. That is a way I am currently trying.
Like Reply privately Flag as inappropriate 11 days ago
Inna Ben-Anat
Inna
Inna Ben-Anat
Director, Global QbD Strategy and Product Robustness at Teva Pharmaceuticals
Hi Sun. The first part of your blog literally had me laughing out loud. This is so true and relevant.
I don’t want to repeat what others and you said, I agree that at initial stage of product development a more simplistic approach works and serve the purpose much better. Traffic light matrix, for example, similarly to what is published in FDA case studies, works really well (also addressing the confusions what is X and what is Y). I think more focus should be put on risk ranking justification, and here is where science and stats can work great together. I like the approach Stefano outlined. Combining scientific rationale of risk level with confirmed data mining output based on historical data of a similar product can really make the risk assessment process value added and focus the development on truly critical and relevant aspects for product CQAs.
Waiting for part II….
Like (1) Reply privately Flag as inappropriate 11 days ago Balaji Somwanshi likes this
Pedro Hernandez
Pedro
Pedro Hernandez
Director of Quality by Design (QbD) at Merck Serono
It should be a fit for purpose process, from qualitative to quantitative, from Ishikawa to FMEA.
Like Reply privately Flag as inappropriate 11 days ago
Stefano Selva
Stefano
Stefano Selva
R&D Pharma Formulation Development at Angelini
Agree with Inna, combining stat/math and prior scientifical knowledge is the key. But..in my opinion another limitation of the FMEA approach is – by definition – that it is an univariate (qualitative or semi-quantitative) tool. How is it possible to assign a correct risk ranking to pharma cPPs using an univariate approach?? Yes, it is useful to list them, but it is a risk to rank them. Formulation cPPs, Equipments cPPs, CMAs, environmental conditions (noises) and bulk variability are ALWAYS (or often) interacting in the pharma field, don’t you agree? Analysing exhisting (and similar) processes/products using MVA is fundamental to highlight these interactions and their impact (if any) to the CQAs. A correct understanding of this help also for the further selection of a correct DoE approach (if interactions are a real challenge for CQAs, it must be considered and good Resolution Fractional design must be chosen). Waiting for part II too..
Like Reply privately Flag as inappropriate 11 days ago
Francesca Speroni
Francesca
Francesca Speroni
Project Leader at PTM Consulting srl
Hi Sun! I’m still laughing after reading your blog… it was a really very interesting and all the comments are very interesting as well.
I completely agree with you that FMEA, especially in the early development, is not the ideal tool but there exist simplest tools that can be used especially when knowledge of product/process is not high… In my experience, the use of FMEA in the early situations if affected by the fact that, since the team does not know a lot of the product, everything seems to be high critical and, in the end, everything results critical. Moreover, if you don’t have experience of the product/process, terms like occurrence and detection are difficult to estimate and, for this reason, the final Risk Index can be misleading. This is not the real meaning of risk assessment… I completely agree that,especially in the early phases the real problem, is the increase in understanding of the product/process in order to really quantify the criticality: in this sense DoE, MVA and other statistical tools are becoming more and more important.
Like Reply privately Flag as inappropriate 11 days ago
Matej Janovjak
Matej
Matej Janovjak
Unternehmensinhaber InSystea GmbH
Dears, the key is process understanding (i.e. impact of process parameters PP & material attributes MA – Variations on product attributes) and knowledge to apply FMEA, then is FMEA “good enough” for univariate risk consideration;
from the perspective of assessment of multivariate risk propagation across the process (partly understood by DOE, MVA etc.) a process & product model (i.e. knowledge of multivariate cause-and-effect relationships between PP, MA and product attributes) is needed as basis, especially in early development.
(if you don’t mind to have a look at http://www.insystea.ch/index.php/en/methoden/kausale-produkt-prozess-darstellung and http://www.insystea.ch/index.php/en/methoden/risiko-management)
Like Reply privately Flag as inappropriate 11 days ago
P Venkat Bhaskar Rao
P Venkat
P Venkat Bhaskar Rao
Manager at Celltrion Chemical Research Institute
Dear Sun Kim,
That was a nice blog, representing a typical scenario, practically.
As mentioned above, it would be a wise approach to simplify risk assessment approach, based on the limited understanding during initial stages of product development.
An efficient tool like FMEA, helps obtain great results, when used with sufficient understanding.
The blog describes a scenario where FMEA was conducted and, concerns were raised on its translation to manufacturing and regulatory burden. In such case, at times to improve understanding one may have to practically take trials on the manufacturing scale, prior to performing FMEA.
Thank you all for the insight.
Like Reply privately Flag as inappropriate 10 days ago
Mehtap Saydam
Mehtap
Mehtap Saydam
R&D Specialist (M Sc Pharm)/ Sanovel
Top Contributor
Hi Sun, Again an excellent blog and excellent discussion!
Definitely for a succesful QbD implementation can not be achieved without a successful risk evaluation. Because there are many kind of risks on pharmaceutical industry like a favorite citation ““In this world nothing can be said to be certain, except death and taxes” Benjamin Franklin.
Generally, moving within broad types of risk assessment from qualitative to quantitative, causes increase on objectivity, and may be little complicates cheating. This is one of the major benefits of risk assessments, since it is normally a subjective notion. On this point FMEA and such like numerical tools can be more objective. Lots of approachs can be applied with such numeric tools, like using batch process control datas, distrubutions or simulation results. You may prevent from “fighting from 3 to 4” by using these scientific statistical datas Three level ranking system can be another very useful approach as “Low-Med-High”. At the same time, these tools can be combined, like creating FTA trees with using FMEA risk numbers for an enriched visualization. At the end of the day selection of the right method from all these kind of applications should depend on science based process needs. Because they are tools for improving the quality, not the aim, like your nice ending of the blog.
In development projects as you mentioned, it is really hard to generate those datas for FMEA. Using historical datas for “similar” product can be useful, but has important circumstances, and needs detailed statistical knowledge and it is a great doubt if regulatory will keen on just like applications. So you may have to change your tool to less objective ones. In recent studies, “Super Fishbone” diagrams taking my interest. For lastly I have one small addition to your blog, about Detectability. Detectability may effect Occuracy. But at the same time it is related with control strategy. As your example, when definition of Risk of Potency, risk may cause from Product itself (Occuracy) or product may not have problem but Assay method may have a difficulty (Detectability). So I think they should considered individually. May be Part II will reply to my comments.
Like Reply privately Flag as inappropriate 10 days ago
piyush
piyush chudiwal
Formulation Development at Dr. Reddy’s Laboratories
Hi sun
This is very often and practical situation for the products which are introduced for development.
For the efficient use of FMEA / FTA tool, we should update it from different stages of development as our scientific domain for particular product increases. At initial stages limited understanding of particular product/process leads to seem everything at high risk (Critical); so the same is misleading. As project moves all input domain (CMA Exp, CMA API, CPP) knowledge increases which reduces risk. So after particular stage of development we should update FMEA (Like Preformulation stage; prototype stage; pilot stage ……)
Like Reply privately Flag as inappropriate 9 days ago
Sun Kim
Sun Kim
Manager, Master Black Belt in Quality-by-Design, Agile Lean Six Sigma and Design Thinking
Top Contributor
Sarah, Stefano, Inna, Pedro, Francesca, Matej, P Venkat, Mehtap, Piyush, Thanks for sharing your experience! I am refining the details now before I can share. Thanks for your patience!
Delete 6 days ago
Alicia Tébar
Alicia
Alicia Tébar
Project manager QA & QbD en Azbil Telstar
Hi peers, completely agree with you. I’ve never used FMEA for development RA, as you would have to mostly “invent” the probability rates for instance. Sometimes you have to fight against the idea that FMEA is in general the most well accepted RA tool among regulatory authorities and that other kind of more “informal” RA approaches are not as consistent. I think that this is maybe true for legacy products but not for new developments.
Much more value from previous/historical data to guide DoE than trying to fit a perfect risk analysis.
Like (1) Reply privately Flag as inappropriate 6 days ago Pedro Hernandez likes this
Matej Horvat
Matej
Matej Horvat
Scientific Advisor at Lek, a Sandoz company
Hi Sun, nice observation about FMEA. Another of my FMEA favorites is the detectability issue. I have seen people in development down-rating a risk by giving it a D = 1, and the risk disappears in the background. Therefore in my company we use a different and more appropriate approach, but I can’t share any details unfortunately.
Best,
Matej
Like Reply privately Flag as inappropriate 1 day ago
Irwin Hirsh
Irwin
Irwin Hirsh
QbD Implementation
Great stuff everyone. Below I will ramble on a bit…sorry if it is obtuse…trying to respond to all
In the case of early product development the FMEA is of course an important tool, but what we are finding (current and previous employer), the use of some tools are not so helpful until the basic product design choices are made. It should be considered that in the absence of QTPP or a good connection between QTPP and CQAs you may have some serious deconvolution issues around Product Specification (design issues) and the prioritization of process parameters that need to be investigated. That is on a high level, have you defined the product feature(s) that will enable you to achieve the label claims you would like to make (TPPs). In the case of API: chemical synthesis it would be “route selection” or for biologic API in biologics (cell line choice, perfusion/continuous vs. batch) and their consequences down the value chain.
In Drug product it would be the rationale that defines how to chose between similar things For example: in a tablet, defining the rational about why dry compression is the best design.
Once these choices are made, or dictated by technology platforms, the process FMEA is really helpful if parameters are well defined and the use of the tool is understood.
We have found it hard to: Include critical-to-business in along with the critical-to-quality focus though both are important to process development, only one of course trumps if need be. ensure that people understand the use of the concept of Failure Mode (also shows up in poor CAPA root cause in mfg. sites everywhere), especially when detectability is invoked. Sure you can detect the failure has occurred but the real question is why did the failure occur. lowering a risk scoring (RPN) mostly due to Detection results in some wonky (low capability) processes and lots of end product testing (quality by inspection)
Another big issue is that a one-off Risk Assessment or even a few rounds with the FMEA is not QRM. The process of QRM must connect across the life stages and with the right stakeholders via participation even if it is only via a good risk communication document. Let alone the use of a previously agreed upon understanding of vocabulary and acceptable risks…
I could probably get the owner to pay a lot more cash to bring in an an early stage product into our pipeline of we had a good presentation of risk (quality and business) during due diligence. Proper understanding of these risk (or best possible at the time) will facilitate any technical transfer or interface….or stop things from moving forward as time-bombs into the later stages
looking forward to comments if any…Irwin
Like Reply privately Flag as inappropriate 1 day ago
Kathryn A. Lee
Kathryn A.
Kathryn A. Lee
Head of the Laboratory at rap.ID Inc.
FMEA should include those with knowledge and experience with all aspects of a process including those involved with raw materials, process engineers, those who run the processes, product developers who should understand the product, analytical scientists, and even perhaps maintenance and site managers. Or have “teams” become unimportant?
Like Reply privately Flag as inappropriate 21 hours ago
David LeBlond
David
David LeBlond
Statistician
Sun et al, How refreshing to see this. Thank you so much for noting that the FMEA emperor may be missing some clothing.
In fact RPN has no mathematical basis. Multiplication is not defined for ordinal metrics. It is well known in the literature that RPN does not preserve rank order. So why not use probability and cost metrics instead of ranks? Doing so won’t simplify negotiations. Prior subjective knowledge (not just data) must be managed and leveraged. Risk assessment by its nature is hard. But the final product will be robust. Below are some references that discuss this:
W. Gilchrist, ?Modelling Failure Modes and Effects Analysis,? International Journal of Quality and Reliability Management 10 (5), 16-23, 1993.
S. Kmenta, Scenario-based FMEA Using Expected Cost, A new perspective on evaluatng risk in FMEA, IIE Workshop,January 22, 2002.
S. Kmenta and K. Ishill, Scenario-Based Failure Modes and Effects Analysis using Expected Costs, Journal of Mechanical Design 126, 1027-1036, 2004.
D. Wheeler, ?Problems with Risk Priority Numbers,Quality Digest, 2011 Available at:
http://www.qualitydigest.com/inside/quality-insider-article/problems-risk-priority-numbers.html
Like Reply privately Flag as inappropriate 18 hours ago
Irwin Hirsh
Irwin
Irwin Hirsh
QbD Implementation
Cross functional Teams…solving problems of space time and busy scheudules
Hi Kathryn
It has been my experience that getting a good team together under one roof is very tough. I have been facilitating as few full team meetings as possible (minimum one plenum session) and then trying to run things “asynchronously” I have teams review the FMEA in pairs or individually and then have a final or time critical group meeting to create a consensus document before actions are taken. this actually seems to work well.
As you seem to see the importance of teams do you have any direct experience with getting everyone together or comments on my approach above. If so please do share…would be great
BR, Irwin
Like Reply privately Flag as inappropriate 12 hours ago
Kathryn A. Lee
Kathryn A.
Kathryn A. Lee
Head of the Laboratory at rap.ID Inc.
Irwin,
Yes, we do have to be creative about how we get and disperse information. I worked for several years with people at plants that I never saw in person. I am an analytical chemist, and people very often submit samples with a question, “What is this? or asking for a particular analytical test, when what they really want is a solution to the problem of why this batch came out wrong and how to prevent that from ever happening. So part of the problem is that they think they will know how to solve the problem when they find out what it is, but in fact, there are often many more subtleties to solving the problem, and other people who could help them better if they just asked.
At one time,there was a problem with production, and all the managers got together to try to figure it out, and after several weeks, they finally asked the people running the process, and the line workers all knew the problem was due to the night shift who didn’t know how to set up the filters properly. Of course the point of that story, which has probably happened lots of times in lots of different ways, is to communicate well with everyone and make sure you respect their knowledge and skill.
Like Reply privately Flag as inappropriate 9 hours ago
piyush
piyush chudiwal
Formulation Development at Dr. Reddy’s Laboratories
Hi sun
This is very often and practical situation for the products which are introduced for development.
For the efficient use of FMEA / FTA tool, we should update it from different stages of development as our scientific domain for particular product increases. At initial stages limited understanding of particular product/process leads to seem everything at high risk (Critical); so the same is misleading. As project moves all input domain (CMA Exp, CMA API, CPP) knowledge increases which reduces risk. So after particular stage of development we should update FMEA (Like Preformulation stage; prototype stage; pilot stage ……)
Like Reply privately Flag as inappropriate 9 days ago
Sun Kim
Sun Kim
Manager, Master Black Belt in Quality-by-Design, Agile Lean Six Sigma and Design Thinking
Top Contributor
Sarah, Stefano, Inna, Pedro, Francesca, Matej, P Venkat, Mehtap, Piyush, Thanks for sharing your experience! I am refining the details now before I can share. Thanks for your patience!
Delete 6 days ago
Alicia Tébar
Alicia
Alicia Tébar
Project manager QA & QbD en Azbil Telstar
Hi peers, completely agree with you. I’ve never used FMEA for development RA, as you would have to mostly “invent” the probability rates for instance. Sometimes you have to fight against the idea that FMEA is in general the most well accepted RA tool among regulatory authorities and that other kind of more “informal” RA approaches are not as consistent. I think that this is maybe true for legacy products but not for new developments.
Much more value from previous/historical data to guide DoE than trying to fit a perfect risk analysis.
Like (1) Reply privately Flag as inappropriate 6 days ago Pedro Hernandez likes this
Matej Horvat
Matej
Matej Horvat
Scientific Advisor at Lek, a Sandoz company
Hi Sun, nice observation about FMEA. Another of my FMEA favorites is the detectability issue. I have seen people in development down-rating a risk by giving it a D = 1, and the risk disappears in the background. Therefore in my company we use a different and more appropriate approach, but I can’t share any details unfortunately.
Best,
Matej
Like Reply privately Flag as inappropriate 1 day ago
Irwin Hirsh
Irwin
Irwin Hirsh
QbD Implementation
Great stuff everyone. Below I will ramble on a bit…sorry if it is obtuse…trying to respond to all
In the case of early product development the FMEA is of course an important tool, but what we are finding (current and previous employer), the use of some tools are not so helpful until the basic product design choices are made. It should be considered that in the absence of QTPP or a good connection between QTPP and CQAs you may have some serious deconvolution issues around Product Specification (design issues) and the prioritization of process parameters that need to be investigated. That is on a high level, have you defined the product feature(s) that will enable you to achieve the label claims you would like to make (TPPs). In the case of API: chemical synthesis it would be “route selection” or for biologic API in biologics (cell line choice, perfusion/continuous vs. batch) and their consequences down the value chain.
In Drug product it would be the rationale that defines how to chose between similar things For example: in a tablet, defining the rational about why dry compression is the best design.
Once these choices are made, or dictated by technology platforms, the process FMEA is really helpful if parameters are well defined and the use of the tool is understood.
We have found it hard to: Include critical-to-business in along with the critical-to-quality focus though both are important to process development, only one of course trumps if need be. ensure that people understand the use of the concept of Failure Mode (also shows up in poor CAPA root cause in mfg. sites everywhere), especially when detectability is invoked. Sure you can detect the failure has occurred but the real question is why did the failure occur. lowering a risk scoring (RPN) mostly due to Detection results in some wonky (low capability) processes and lots of end product testing (quality by inspection)
Another big issue is that a one-off Risk Assessment or even a few rounds with the FMEA is not QRM. The process of QRM must connect across the life stages and with the right stakeholders via participation even if it is only via a good risk communication document. Let alone the use of a previously agreed upon understanding of vocabulary and acceptable risks…
I could probably get the owner to pay a lot more cash to bring in an an early stage product into our pipeline of we had a good presentation of risk (quality and business) during due diligence. Proper understanding of these risk (or best possible at the time) will facilitate any technical transfer or interface….or stop things from moving forward as time-bombs into the later stages
looking forward to comments if any…Irwin
Like Reply privately Flag as inappropriate 1 day ago
Kathryn A. Lee
Kathryn A.
Kathryn A. Lee
Head of the Laboratory at rap.ID Inc.
FMEA should include those with knowledge and experience with all aspects of a process including those involved with raw materials, process engineers, those who run the processes, product developers who should understand the product, analytical scientists, and even perhaps maintenance and site managers. Or have “teams” become unimportant?
Like Reply privately Flag as inappropriate 21 hours ago
David LeBlond
David
David LeBlond
Statistician
Sun et al, How refreshing to see this. Thank you so much for noting that the FMEA emperor may be missing some clothing.
In fact RPN has no mathematical basis. Multiplication is not defined for ordinal metrics. It is well known in the literature that RPN does not preserve rank order. So why not use probability and cost metrics instead of ranks? Doing so won’t simplify negotiations. Prior subjective knowledge (not just data) must be managed and leveraged. Risk assessment by its nature is hard. But the final product will be robust. Below are some references that discuss this:
W. Gilchrist, ?Modelling Failure Modes and Effects Analysis,? International Journal of Quality and Reliability Management 10 (5), 16-23, 1993.
S. Kmenta, Scenario-based FMEA Using Expected Cost, A new perspective on evaluatng risk in FMEA, IIE Workshop,January 22, 2002.
S. Kmenta and K. Ishill, Scenario-Based Failure Modes and Effects Analysis using Expected Costs, Journal of Mechanical Design 126, 1027-1036, 2004.
D. Wheeler, ?Problems with Risk Priority Numbers,Quality Digest, 2011 Available at:
http://www.qualitydigest.com/inside/quality-insider-article/problems-risk-priority-numbers.html
Like Reply privately Flag as inappropriate 18 hours ago
Irwin Hirsh
Irwin
Irwin Hirsh
QbD Implementation
Cross functional Teams…solving problems of space time and busy scheudules
Hi Kathryn
It has been my experience that getting a good team together under one roof is very tough. I have been facilitating as few full team meetings as possible (minimum one plenum session) and then trying to run things “asynchronously” I have teams review the FMEA in pairs or individually and then have a final or time critical group meeting to create a consensus document before actions are taken. this actually seems to work well.
As you seem to see the importance of teams do you have any direct experience with getting everyone together or comments on my approach above. If so please do share…would be great
BR, Irwin
Like Reply privately Flag as inappropriate 12 hours ago
Kathryn A. Lee
Kathryn A.
Kathryn A. Lee
Head of the Laboratory at rap.ID Inc.
Irwin,
Yes, we do have to be creative about how we get and disperse information. I worked for several years with people at plants that I never saw in person. I am an analytical chemist, and people very often submit samples with a question, “What is this? or asking for a particular analytical test, when what they really want is a solution to the problem of why this batch came out wrong and how to prevent that from ever happening. So part of the problem is that they think they will know how to solve the problem when they find out what it is, but in fact, there are often many more subtleties to solving the problem, and other people who could help them better if they just asked.
At one time,there was a problem with production, and all the managers got together to try to figure it out, and after several weeks, they finally asked the people running the process, and the line workers all knew the problem was due to the night shift who didn’t know how to set up the filters properly. Of course the point of that story, which has probably happened lots of times in lots of different ways, is to communicate well with everyone and make sure you respect their knowledge and skill.
Comments from LinkedIn were valuable so i echoed them here.
A little late to get here, it seems.
This is a good blog!
Any Risk Management activity starts with ‘Risk Identification’. FMEA is NOT a risk idetification tool. Using FMEA at a preliminary stage in development may result in inflated RPNs to start with as you mention. But there is another hazard: a drastically low RPN at the end of development! Because every stage in development may be motivated by a reduction in RPN. To the point that one has very little residual risk before manufacturing- which is completely misleading. I agree with Inna here: the traffic light risk index works wonders as long as there is a scientific judgement behind the colors.
Jasmine, great comments! I think it’s time we explore better tools to tie risk assessment and management. Please share more of your experience.
Updates on comments from the QbD Linkedin community.
Sarah Betterman
Sarah
Sarah Betterman
Quality by Design │ Lean Six Sigma Black Belt
I agree with your perspective, Sun. The formality of risk assessment should align with the level of product and process understanding. FMEA definitely has its place, but later in the development process. To simplify even further, I have used a two-level Low-High ranking system. However, I look forward to hearing the details of your approach as I am always looking for ways to improve!
Like (1) Reply privately Flag as inappropriate 1 month ago Line Lundsberg-Nielsen likes this
Stefano Selva
Stefano
Stefano Selva
Senior Scientist Formulation – Pharmaceutical Sciences at Aptuit
Great discussion! It is like a loop I always have to dealt with, everytime! The importance of the FMEA approach is surely not under discussion. Despite of this, in my opinion, relying just on the FMEA approach in the early-medium stages of development (prior of scale up) could bring to serious risk (it is like a ‘bias’ related to the scientists’ prior knowledge). Using an approch consisting on data mining / ‘passive’ multivariate analysis of ‘previous and similar formulation and processes’ (a simple PCA for example?) could be, in my opinion, a good way to asses CPPs and Critical parameters of raw materials (and bulk variability!). This analysis (based on real/process data and products) could help defining the most significant factors to study on a DoE for new developments. That is a way I am currently trying.
Like Reply privately Flag as inappropriate 1 month ago
Inna Ben-Anat
Inna
Inna Ben-Anat
Director, Global QbD Strategy and Product Robustness at Teva Pharmaceuticals
Hi Sun. The first part of your blog literally had me laughing out loud. This is so true and relevant.
I don’t want to repeat what others and you said, I agree that at initial stage of product development a more simplistic approach works and serve the purpose much better. Traffic light matrix, for example, similarly to what is published in FDA case studies, works really well (also addressing the confusions what is X and what is Y). I think more focus should be put on risk ranking justification, and here is where science and stats can work great together. I like the approach Stefano outlined. Combining scientific rationale of risk level with confirmed data mining output based on historical data of a similar product can really make the risk assessment process value added and focus the development on truly critical and relevant aspects for product CQAs.
Waiting for part II….
Unlike Reply privately Flag as inappropriate 1 month ago Sun Kim, Balaji Somwanshi like this
Pedro Hernandez
Pedro
Pedro Hernandez
Director of Quality by Design (QbD) at Merck Serono
It should be a fit for purpose process, from qualitative to quantitative, from Ishikawa to FMEA.
Like Reply privately Flag as inappropriate 1 month ago
Stefano Selva
Stefano
Stefano Selva
Senior Scientist Formulation – Pharmaceutical Sciences at Aptuit
Agree with Inna, combining stat/math and prior scientifical knowledge is the key. But..in my opinion another limitation of the FMEA approach is – by definition – that it is an univariate (qualitative or semi-quantitative) tool. How is it possible to assign a correct risk ranking to pharma cPPs using an univariate approach?? Yes, it is useful to list them, but it is a risk to rank them. Formulation cPPs, Equipments cPPs, CMAs, environmental conditions (noises) and bulk variability are ALWAYS (or often) interacting in the pharma field, don’t you agree? Analysing exhisting (and similar) processes/products using MVA is fundamental to highlight these interactions and their impact (if any) to the CQAs. A correct understanding of this help also for the further selection of a correct DoE approach (if interactions are a real challenge for CQAs, it must be considered and good Resolution Fractional design must be chosen). Waiting for part II too..
Like Reply privately Flag as inappropriate 1 month ago
Francesca Speroni
Francesca
Francesca Speroni
Project Leader at PTM Consulting srl
Hi Sun! I’m still laughing after reading your blog… it was a really very interesting and all the comments are very interesting as well.
I completely agree with you that FMEA, especially in the early development, is not the ideal tool but there exist simplest tools that can be used especially when knowledge of product/process is not high… In my experience, the use of FMEA in the early situations if affected by the fact that, since the team does not know a lot of the product, everything seems to be high critical and, in the end, everything results critical. Moreover, if you don’t have experience of the product/process, terms like occurrence and detection are difficult to estimate and, for this reason, the final Risk Index can be misleading. This is not the real meaning of risk assessment… I completely agree that,especially in the early phases the real problem, is the increase in understanding of the product/process in order to really quantify the criticality: in this sense DoE, MVA and other statistical tools are becoming more and more important.
Like Reply privately Flag as inappropriate 1 month ago
Matej Janovjak
Matej
Matej Janovjak
Unternehmensinhaber InSystea GmbH
Dears, the key is process understanding (i.e. impact of process parameters PP & material attributes MA – Variations on product attributes) and knowledge to apply FMEA, then is FMEA “good enough” for univariate risk consideration;
from the perspective of assessment of multivariate risk propagation across the process (partly understood by DOE, MVA etc.) a process & product model (i.e. knowledge of multivariate cause-and-effect relationships between PP, MA and product attributes) is needed as basis, especially in early development.
(if you don’t mind to have a look at http://www.insystea.ch/index.php/en/methoden/kausale-produkt-prozess-darstellung and http://www.insystea.ch/index.php/en/methoden/risiko-management)
Like Reply privately Flag as inappropriate 1 month ago
P Venkat Bhaskar Rao
P Venkat
P Venkat Bhaskar Rao
Manager at Celltrion Chemical Research Institute
Dear Sun Kim,
That was a nice blog, representing a typical scenario, practically.
As mentioned above, it would be a wise approach to simplify risk assessment approach, based on the limited understanding during initial stages of product development.
An efficient tool like FMEA, helps obtain great results, when used with sufficient understanding.
The blog describes a scenario where FMEA was conducted and, concerns were raised on its translation to manufacturing and regulatory burden. In such case, at times to improve understanding one may have to practically take trials on the manufacturing scale, prior to performing FMEA.
Thank you all for the insight.
Like Reply privately Flag as inappropriate 1 month ago
Mehtap Saydam
Mehtap
Mehtap Saydam
R&D Specialist (M Sc Pharm)/ Sanovel
Hi Sun, Again an excellent blog and excellent discussion!
Definitely for a succesful QbD implementation can not be achieved without a successful risk evaluation. Because there are many kind of risks on pharmaceutical industry like a favorite citation ““In this world nothing can be said to be certain, except death and taxes” Benjamin Franklin.
Generally, moving within broad types of risk assessment from qualitative to quantitative, causes increase on objectivity, and may be little complicates cheating. This is one of the major benefits of risk assessments, since it is normally a subjective notion. On this point FMEA and such like numerical tools can be more objective. Lots of approachs can be applied with such numeric tools, like using batch process control datas, distrubutions or simulation results. You may prevent from “fighting from 3 to 4” by using these scientific statistical datas Three level ranking system can be another very useful approach as “Low-Med-High”. At the same time, these tools can be combined, like creating FTA trees with using FMEA risk numbers for an enriched visualization. At the end of the day selection of the right method from all these kind of applications should depend on science based process needs. Because they are tools for improving the quality, not the aim, like your nice ending of the blog.
In development projects as you mentioned, it is really hard to generate those datas for FMEA. Using historical datas for “similar” product can be useful, but has important circumstances, and needs detailed statistical knowledge and it is a great doubt if regulatory will keen on just like applications. So you may have to change your tool to less objective ones. In recent studies, “Super Fishbone” diagrams taking my interest. For lastly I have one small addition to your blog, about Detectability. Detectability may effect Occuracy. But at the same time it is related with control strategy. As your example, when definition of Risk of Potency, risk may cause from Product itself (Occuracy) or product may not have problem but Assay method may have a difficulty (Detectability). So I think they should considered individually. May be Part II will reply to my comments.
Like Reply privately Flag as inappropriate 1 month ago
piyush
piyush chudiwal
Formulation Development at Dr. Reddy’s Laboratories
Hi sun
This is very often and practical situation for the products which are introduced for development.
For the efficient use of FMEA / FTA tool, we should update it from different stages of development as our scientific domain for particular product increases. At initial stages limited understanding of particular product/process leads to seem everything at high risk (Critical); so the same is misleading. As project moves all input domain (CMA Exp, CMA API, CPP) knowledge increases which reduces risk. So after particular stage of development we should update FMEA (Like Preformulation stage; prototype stage; pilot stage ……)
Like Reply privately Flag as inappropriate 1 month ago
Sun Kim
Sun Kim
Sr. Manager, Master Black Belt in Quality-by-Design, Agile Lean Six Sigma and Design Thinking
Sarah, Stefano, Inna, Pedro, Francesca, Matej, P Venkat, Mehtap, Piyush, Thanks for sharing your experience! I am refining the details now before I can share. Thanks for your patience!
Delete 1 month ago
Alicia Tébar
Alicia
Alicia Tébar
Project manager QA & QbD en Azbil Telstar
Hi peers, completely agree with you. I’ve never used FMEA for development RA, as you would have to mostly “invent” the probability rates for instance. Sometimes you have to fight against the idea that FMEA is in general the most well accepted RA tool among regulatory authorities and that other kind of more “informal” RA approaches are not as consistent. I think that this is maybe true for legacy products but not for new developments.
Much more value from previous/historical data to guide DoE than trying to fit a perfect risk analysis.
Unlike Reply privately Flag as inappropriate 1 month ago Sun Kim, Pedro Hernandez and 1 other like this
Matej Horvat
Matej
Matej Horvat
Scientific Advisor at Lek, a Sandoz company
Top Contributor
Hi Sun, nice observation about FMEA. Another of my FMEA favorites is the detectability issue. I have seen people in development down-rating a risk by giving it a D = 1, and the risk disappears in the background. Therefore in my company we use a different and more appropriate approach, but I can’t share any details unfortunately.
Best,
Matej
Unlike Reply privately Flag as inappropriate 1 month ago Sun Kim likes this
Irwin Hirsh
Irwin
Irwin Hirsh
QbD Implementation
Great stuff everyone. Below I will ramble on a bit…sorry if it is obtuse…trying to respond to all
In the case of early product development the FMEA is of course an important tool, but what we are finding (current and previous employer), the use of some tools are not so helpful until the basic product design choices are made. It should be considered that in the absence of QTPP or a good connection between QTPP and CQAs you may have some serious deconvolution issues around Product Specification (design issues) and the prioritization of process parameters that need to be investigated. That is on a high level, have you defined the product feature(s) that will enable you to achieve the label claims you would like to make (TPPs). In the case of API: chemical synthesis it would be “route selection” or for biologic API in biologics (cell line choice, perfusion/continuous vs. batch) and their consequences down the value chain.
In Drug product it would be the rationale that defines how to chose between similar things For example: in a tablet, defining the rational about why dry compression is the best design.
Once these choices are made, or dictated by technology platforms, the process FMEA is really helpful if parameters are well defined and the use of the tool is understood.
We have found it hard to: Include critical-to-business in along with the critical-to-quality focus though both are important to process development, only one of course trumps if need be. ensure that people understand the use of the concept of Failure Mode (also shows up in poor CAPA root cause in mfg. sites everywhere), especially when detectability is invoked. Sure you can detect the failure has occurred but the real question is why did the failure occur. lowering a risk scoring (RPN) mostly due to Detection results in some wonky (low capability) processes and lots of end product testing (quality by inspection)
Another big issue is that a one-off Risk Assessment or even a few rounds with the FMEA is not QRM. The process of QRM must connect across the life stages and with the right stakeholders via participation even if it is only via a good risk communication document. Let alone the use of a previously agreed upon understanding of vocabulary and acceptable risks…
I could probably get the owner to pay a lot more cash to bring in an an early stage product into our pipeline of we had a good presentation of risk (quality and business) during due diligence. Proper understanding of these risk (or best possible at the time) will facilitate any technical transfer or interface….or stop things from moving forward as time-bombs into the later stages
looking forward to comments if any…Irwin
Like (2) Reply privately Flag as inappropriate 1 month ago Jesper Thorsen, Pierre Lepage like this
Kathryn A. Lee
Kathryn A.
Kathryn A. Lee
Lab Head at rap.ID Inc.; Raman, IR Spectroscopist
FMEA should include those with knowledge and experience with all aspects of a process including those involved with raw materials, process engineers, those who run the processes, product developers who should understand the product, analytical scientists, and even perhaps maintenance and site managers. Or have “teams” become unimportant?
Like (1) Reply privately Flag as inappropriate 1 month ago Pushpa Tiwari likes this
David LeBlond
David
David LeBlond
Statistician
Sun et al, How refreshing to see this. Thank you so much for noting that the FMEA emperor may be missing some clothing.
In fact RPN has no mathematical basis. Multiplication is not defined for ordinal metrics. It is well known in the literature that RPN does not preserve rank order. So why not use probability and cost metrics instead of ranks? Doing so won’t simplify negotiations. Prior subjective knowledge (not just data) must be managed and leveraged. Risk assessment by its nature is hard. But the final product will be robust. Below are some references that discuss this:
W. Gilchrist, ?Modelling Failure Modes and Effects Analysis,? International Journal of Quality and Reliability Management 10 (5), 16-23, 1993.
S. Kmenta, Scenario-based FMEA Using Expected Cost, A new perspective on evaluatng risk in FMEA, IIE Workshop,January 22, 2002.
S. Kmenta and K. Ishill, Scenario-Based Failure Modes and Effects Analysis using Expected Costs, Journal of Mechanical Design 126, 1027-1036, 2004.
D. Wheeler, ?Problems with Risk Priority Numbers,Quality Digest, 2011 Available at:
http://www.qualitydigest.com/inside/quality-insider-article/problems-risk-priority-numbers.html
Unlike Reply privately Flag as inappropriate 1 month ago Sun Kim likes this
Irwin Hirsh
Irwin
Irwin Hirsh
QbD Implementation
Cross functional Teams…solving problems of space time and busy scheudules
Hi Kathryn
It has been my experience that getting a good team together under one roof is very tough. I have been facilitating as few full team meetings as possible (minimum one plenum session) and then trying to run things “asynchronously” I have teams review the FMEA in pairs or individually and then have a final or time critical group meeting to create a consensus document before actions are taken. this actually seems to work well.
As you seem to see the importance of teams do you have any direct experience with getting everyone together or comments on my approach above. If so please do share…would be great
BR, Irwin
Unlike Reply privately Flag as inappropriate 1 month ago Sun Kim likes this
Kathryn A. Lee
Kathryn A.
Kathryn A. Lee
Lab Head at rap.ID Inc.; Raman, IR Spectroscopist
Irwin,
Yes, we do have to be creative about how we get and disperse information. I worked for several years with people at plants that I never saw in person. I am an analytical chemist, and people very often submit samples with a question, “What is this? or asking for a particular analytical test, when what they really want is a solution to the problem of why this batch came out wrong and how to prevent that from ever happening. So part of the problem is that they think they will know how to solve the problem when they find out what it is, but in fact, there are often many more subtleties to solving the problem, and other people who could help them better if they just asked.
At one time,there was a problem with production, and all the managers got together to try to figure it out, and after several weeks, they finally asked the people running the process, and the line workers all knew the problem was due to the night shift who didn’t know how to set up the filters properly. Of course the point of that story, which has probably happened lots of times in lots of different ways, is to communicate well with everyone and make sure you respect their knowledge and skill.
Like (1) Reply privately Flag as inappropriate 1 month ago Pierre Lepage likes this
Sun Kim
Sun Kim
Sr. Manager, Master Black Belt in Quality-by-Design, Agile Lean Six Sigma and Design Thinking
David, thanks for the reference publications! Kmenta and we did our PhD at the same Ishii lab and of course Wheeler is a classic.
I share Irwin’s approach. I call it the hybrid “Nemawashi” approach. No need to conduct big meetings (unless for an important decision) frequently in this day and age with the technology. It’s inefficient and only adds grumpiness. An upcoming post will share details on how to push risk assessment forward “without being delayed by everyone’s schedules.”
Delete 1 month ago
Emil Ciurczak
Emil
Emil Ciurczak
Independent Pharmaceuticals Professional
I agree with Kathryn (full disclosure: she is a friend). Each group tends to treat each other group as “servicemen.” They tend to think that “those” people don’t understand what we’re doing, so why explain in detail? As she mentioned, the plant will send a sample for some assay without explaining why they need it. It may turn out to be something completely different, which could be remedied sooner, IF all the information and the situation were shared.
As foe risk assessment, these same assumptions are made, without asking or explaining and then we wonder where the analysis went wrong. To quote the warden in Cool Hand Luke, “What we have here is failure to communicate.”
Like Reply privately Flag as inappropriate 1 month ago
Jesper Thorsen
Jesper
Jesper Thorsen
Principal Development Scientist at Fertin Pharma A/S
This topic really is important – the key topic in QbD as I see it. Totally agree, not just with you Sun but in general with all comments. And Sun, you made me laugh too: Been there, done that 🙂
What I tried to do is to start by focus on the quality attributes defined in the QTPP for the product you intend end out with. All quality attributes should be ensured during development, and the critical ones (CQAs) are simply those with impact on safety/efficacy – explained by a justification (in most cases these would typically be assay, homogeneity (CU), dissolution, degradation/impurities). But assessed without a quantitative scoring – just by a short justification (as in the ANDA FDA examples). Those not impacting safety/effcacy, e.g. impacting consumer acceptance (for a chewable tablet this could be taste, appearance, hardness etc) are not defined as CQAs but remain (important) quality attributes to be ensured to have a quality product.
Then I start the FMEA, but if you have not yet entered a late stage/up-scale state (or don’t have a platform/prior knowlegde from similar products), I use a “qualitative” FMEA. So in early stage, I list all INPUT formulation and process variables (material attributes and process parameters for all process steps) and just ask myself: Is this MA/PP really a (potential) risk to impact the critical OUTPUT (= the CQAs)? E.g. for the process parameter blending time in a powder blending process step, I would overall just say: YES. But at this point I don’t spend a lot of time to explain this by quantitative numbers on severity, frequency, detectability – I leave them empty. I would just use an overall justification why blending time is important in order to achieve a homogeneous blend since in-sufficient BU impacts the DP CQA of CU and hence this is the reason that this is a potential high risk. And a high risk MA/PP should even 1) be examined to find out, if this was actually a true assessment (= was it really high risk – was the severity was assessed too high due to lack of knowledge) and/or 2) find an appropriate control strategy to mitigate the risk (= lower the scores on frequency and/or detactability – severity cannot be lowered by a control strategy since it is still as severe if it happens).
The link to define this to CMAs and CPPs I find quite “easy”. The reason is, that the first part of the risk assessment (doesn’t matter if the FMEA is in a qualitative or quantitative stage) is based on the risk on CQA impact WITHOUT CONSIDERING a control strategy. Then high risk = critical and is mandatory to have a control strategy to mitigate it to low risk (or require an investigation to see if the true risk on CQA was actually high). If the risk BEFORE CONSIDERING a control strategy was assessed as high (towards impact on CQAs) and when CONSIDERING a control strategy it is assessed as low risk, then you have a CMA/CPP with a suitable (but required) control strategy. The control strategies on CMAs/CPPs will during the life-cycle go to be “more and more validated” from a status of “proposed/defined” control strategies to reach a status of “implemented” control strategies. CPPs should be addressed during PPQ/CPV (and the way these a “proved” to be suitable control strategies to a validated state – this is another but interesting topic :-)).
Note: the quantitative FMEA I would introduce at some point where enough knowledge about product and process is reached. This could for example be in connection with full scale pivotal batches (not much later though).
Irwin: Critical business “parameters” I have tried to add in the same risk assessment. So I have CQAs, non-CQAs (quality attributes not defined as critical) and business (e.g. cost, COGS, man hours, storage time etc) and it can be done if you just are clear in your control strategy, what they support (CQAs, non-CQAs or business).
Like (1) Reply privately Flag as inappropriate 19 days ago Manuel Vega Zepeda likes this
Frank
Frank Koppenhagen
Senior Director of Respiratory Product Development at Lupin Pharmaceuticals
I think many things can be done to make the FMEA process easier and a lot of these factors have nothing to do with the actual FMEA, most of it comes down to preparation and team work. If the FMEA leader on a whim has to get a number of “experts” into a room so they can quickly fill in the numbers, has to back-fill secondary experts for those primary experts who are out of the office, the described scenario will definitely be the result. I think the following is important:
1. Find contributors that are real experts. Don’t assume that because someone is on a project team they are the expert in the field and understand the process. Institutional knowledge and process understanding is typically not with team members who were selected because they are team players, but with individual contributors who live on the fringes of teams. (they tend not to be team-players so are not on teams. How do we keep them involved and transfer their knowledge?) Make sure that everyone is available and is focused. If contributors do not show up, cancel the session. Ideally, take them offsite and remove their phones and computers so they have no distractions.
2. Have a briefing session to talk about the purpose of FMEA. Explain that it is not intended to score every possible scenario that they can come up with (and they will, they are scientists who like science-fiction and will come up with alien invasions as serious risks…) but to identify real-world risks that are supported by data and real-life experience. Making up a risk does not make it real.
3. Use of the 0,1,3,9 scoring system is not intuitive for many.Use the 1,2,3,4 scoring system or zero/low/med/high but afterwards weigh it as 0,1,3,9 to really bring out the high risks.
4. Spend time to quantitate what each score means and write it down. Nothing is worse than changing goalposts during FMEA. Example: Occurrence 1 means between once a month and once a quarter, or between 1 in a million and one in 5 million parts. Do this for each factor. Make a nice table of the scoring system and have it on the table so people can refer to it for each discussion. Whenever there is discussion ask them – so, you are saying this may occur more than once a month? etc. Having a good quantitation of the risks is 90% of the FMEA. It does not have to be linear, and if possible the risks/defect rates should be relevant to the company and in line with what is experienced on current processes. One of the reasons why every factor becomes a high risk is because the participants don’t want to accept any risk and scale the scoring system in such a way that even currently acceptable risks/defect levels are scored as high.
5. Be in-tune with the group. People get tired and want to get out of a long session. Break up the sessions with ‘fun’ activities to get the brains back into gear. Do other team activities in between that do not require thinking about the process to relax the brain. Have fun little gadgets to keep them focused.
6. A lot of the above begs the question: is the leader of the FMEA activity the right person? Is he/she a facilitator, a leader, someone who is generally admired and whose authority is accepted by the team? Someone who can make decisions if the team is stuck only to come back to them later for more input? Someone who is in tune with the team dynamics and knows when it is time to refocus the brains?
I agree FMEA can be painful and frustrating, but with some pre-work and good team leading skills it can be made workable.
Like (1) Reply privately Flag as inappropriate 19 days ago Jesper Thorsen likes this
Sun Kim
Sun Kim
Sr. Manager, Master Black Belt in Quality-by-Design, Agile Lean Six Sigma and Design Thinking
Good remarks, Frank. As you described, I wish every facilitator had the skills and the authority to request enough time and people for these activities. Sadly, it’s a rare luxury in reality.
My opinion was that the output of a Risk Assessment should not vary so much on the Operator (facilitator in this case). Instead of replacing the operator, perhaps we could focus on improving or upgrading the tool/system (FMEA in this case)? 🙂
Delete 18 days ago
Paul
Paul Slaman
Process Engineer at Alexion Pharmaceuticals
I am looking forward to seeing your better way of performing FMEA’s.
I have very rarely seen them done correctly.
Like Reply privately Flag as inappropriate 18 days ago
Ivan Pogorelic
Ivan
Ivan Pogorelic
Associate Director, SeniorTechnical Project Leader, CMC leader at Actelion
How true this blog article is, like many folks here I could not stop laughing..:-) :-)…..And how good the discussion that followed. Some thoughts on the topic. Having all those reds screaming out of our RA template does not necessarily mean we’re on the wrong track. As somebody here mentioned it is very much linked to the level of process knowledge and understanding the team has at a particular point of development trajectory. If we are early and our knowledge is limited I guess there will be a lot of high risks simply because our knowledge is limited. And that’s absolutely OK. We will change those traffic lights once we conduct our experiments and maybe dig deeper into the platforms and historical knowledge. Upon scale up and towards late stage development some of those greens will become reds again and vice versa. It’s not a one-off thing, rather a dynamic living and iterative process. And I also believe the “keep it simple” approach should work at an earlier development stage. I am sure there are bigger issues to solve than struggling for weeks with 3 vs. 3.5 for probability of detection 🙂 As they say, sometimes less is more.
Like (1) Reply privately Flag as inappropriate 17 days ago Jesper Thorsen likes this
Sun Kim
Sun Kim
Sr. Manager, Master Black Belt in Quality-by-Design, Agile Lean Six Sigma and Design Thinking
Ivan, you are so insightful. In fact, I use the same approach in using Risk as the KPI for showing progress to the management that we are improving — by resolving and lowering risks (turning reds into green).
Delete 16 days ago
Matej Horvat
Matej
Matej Horvat
Scientific Advisor at Lek, a Sandoz company
Top Contributor
Sometimes I joke that the best way to bring people up to speed with quality risk assessments is to put them through a few FMEA sessions (and pretend that that’s the best way to do it and everybody does it like that and if they don’t get it it’s their problem), and only THEN offer them better or more suitable alternatives for their particular case.
Otherwise, regardless of what you, do they will first shoot the messenger…
🙂
Regards,
Matej
Like Reply privately Flag as inappropriate 2 days ago
This task of QbD and risk assessments often ends in the use of FMEAs for both development and process risk assessments. They also typically end in frustration for the facilitator and its participants. However, with the correct SOPs and training the frustration can be avoided!
I have extensive experience at risk assessment in the commercial setting as the former trainer at Novartis for risk assessments and FMEAs.
The FMEAs at the development stage as mentioned earlier in this blog are difficult because nearly everything has a severity that is critical and the detections are subjective at this point in the game. The process is the same for both development FMEAs and process FMEAs (when much more about the commercial process has been formalized), however, the rating scales are different. This is the only way that you can be successful at the development state. In fact, removing the detectability rating is a good suggestion if the product/process is in the stage of product development. Its appropriate to bring it into the equation after the product business case is found viable and proof of process in the pilot lab has been completed. It is key to have your quality SOP for risk assessment address both stages of product development as different RA rating scales as not to confuse the facilitators/participants in the process.
Great discussion.
Travis Buel
Manufacturing Engineering Manager – Iroko Pharmaceuticals