Analytical Validation-A House of Cards

E. Dennis Bashaw, Pharm.D

Disclaimer: The presentation today should not be considered, in whole or in part as being statements of policy or recommendation by the US Food and Drug Administration.

The examples given in the presentation today are based on actual situations seen by the presenter. All identifying information has been removed to protect the confidentiality of the applicants involved.

Trends in Drug Discovery

Scannell, JW, et al “Diagnosing the decline in pharmaceutical R&D efficiency” Nature Reviews Drug Discovery, 11:191-200 (March 2012)

4

Between the birth of the world and 2003, there were five exabytes of information created. We [now] create five exabytes every two days. See why it's so painful to operate in information markets?“

Eric Schmidt, Google

5 exabytes of information is the equivalent of 250,000 years of DVD quality video

A 21st Century Perspective on Information

5

An Explosion of Information Yields an Explosion of Questions

“You might very well think that; I could not possibly comment!”

House of CardsIf you do not have confidence in the validation of the analytical method, how can you have confidence in the concentration values?

If you do not have confidence in the concentration values, how can you have confidence in the derived pharmacokinetic parameters?

Safety & Efficacy

Clinical Pharmacology

Bioanalytical Validation

Relevant FDA Guidances

The Importance of Methods Validation to Clinical Pharmacology

Bioanalytical Methods Validation Guidance, pg 2

MDS Canada

Cetero

Bioanalytical Review Problems

Mis-labeled samples Improper Shipping Flawed Extraction

Analytical Problems Calculation Issues Analysis/Reporting

Case Study #1

The Case of the “Come As You Are”

Standard Curve

Case Study #1

IV drug, relative bioavailability study in patients with varying degrees of renal insufficiency

Analytical plan called for daily standard curves to be constructed at 2, 5, 10, 25, 100, 500, and 1000ng/ml using triplicate samples.

Analysis consisted of 5 runs over the course of 3 weeks by the same analyst

Case Study #1Target (ng/mL) 2 5 10 25 100 500 1000

Run 1 1.76 4.2 10.2 22.3 120 515 1100

1.5 5.13 8.7 24.11 111 517.2 1070

2.13 4.8 11.4 27.2 102 505 1012

Mean 1.84 4.78 10.08 24.65 108.25 509.3 1045.5

Run 3 1.55 1005

2.42 1089

2.59 1070

Mean 2.19 1054.67

Run 4 1.76 4.76 10.35 21.45 114 535 1093

1.23 4.22 11.2 22.76 110 503 1125

1.16 3.98 10.22 31.22 102 487 1200

1.94 4.25 27.86 99 493 1103

1.87 5.15 22.4 1114

5.6 26.4

4.9

5.55

Mean 1.59 4.80 10.59 25.35 106.25 504.5 1127

NO DATA !

TRIPLICATE SAMPLES???

Case Study #1Analyst chose the number of standards he

would use to construct a standard curve differently between each assay run.

Worst case was two samples (low & high)!Assay was essentially out of controlOver 5 runs over two weeks no two

“standard” curves were constructed the same way

Report was “signed-off” by analyst, lab supervisor, Director of Analytical Services, and Vice-President for Pharmaceutics!

Case Study #2

The Case of History Repeating Itself

Case Study #2 Initial assay developed in the mid 1980s using, for the time,

state of the art HPLC systemOriginal assay validation report showed adequate accuracy,

precision, sensitivity, selectivity, etc.Drug approved, but due to poor absorption the sponsor

immediately began a series of formulation studies to improve bioavailability

Assay validation of the later studies was merely a copy of the earlier report, a table of standard curve results but no tracings.

Case Study #2Why didn’t the sponsor submit individual study validation

reports or tracings?◦ A. They didn’t have them◦ B. The assay conditions had changed ◦ C. The assay performance had changed

Due to changes in equipment/column the retention time for the parent had gone from 3 minutes to 10 minutes. While this is certainly manageable, the sponsor decided to not disclose this.

Case Study #3

The Case of the “U” Shaped

Concentration Time Curve

Case Study #3“U” Shaped Plasma Concentrations(mean of 12 subjects)

Not surprisingly, the calculation of half-life was “difficult” for Group 2!

Case Study #3In two studies done in support of this project more than a quarter of the subjects in a treatment leg in both trials (assayed at the same time, by the same analyst) showed these results. The study site chose to accept the half-life estimate without comment!

What this is, is the “black box phenomena” of data collection and analysis.

A computer gets the output from the detector, runs the statistical & pharmacokinetic analysis modules, produces standard tables that a report is written from. But the report writer is not necessarily exposed to the primary data.

Study A

Study B

Case Study #4

The Case of “The Missing Reagent”

Case Study #4A study was conducted in 24 subjects 12 normal and 12 with

renal insufficiency.Due to irreversible protein binding, the extracted plasma

samples had to be acidified with 0.1N HCl within an hour of extraction.

Because of analytical problems at their prior lab, the company elected to send their samples to Europe for analysis.

A total of approximately 320 samples were shipped.

Case Study #4Upon analysis, out of 300+ samples

all concentrations were BLOQ!Subsequent investigation revealed

that the SOP did not specify an acidification target and that the stock bottle of HCl was sub-potent (age) and contaminated.

The cost of a bottle of acid and a proper SOP was approximately $1 Million and six months of development time.

Lessons

Case #1-Written SOPs are only effective when followed. The fact that an analyst could change procedures on a daily basis and yet all levels of management signed off on the report should not be possible.

Case #2-Analytical methods need to be constantly monitored for changes in performance and when found must be investigated. Hoping the FDA will not ask questions is not a risk management strategy.

Lessons

Case #3-Data analysis should include a program of primary data examination. Over-reliance on the computer to catch errors is totally dependent upon an exhaustive programming of failure modes and is unlikely to ever be all inclusive.

Case #4-Common laboratory reagents play a key role in analysis. A seemingly small detail caused a costly delay. Does your lab have an SOP on reagents and how well is it followed? Last inventory?

Parallels From Other Industries

NASA prepares a monthly safety case study that looks at system failures of many kinds.

August 2012 issue dealt with the loss of Air France Flight 447

The plane was lost with all hands due to a combination of design and a lack of understanding of failure modes by the crew that led to inappropriate control inputs.

The “Glass Cockpit” ProblemThe ability to automation controls during flight is a huge benefit to the safety of the flight during normal situations. However, there are some very big problems that crop up.

Automation Bias: This is where the pilots use the automation, such as auto pilot, as substitute to the gathering of information. They lose situational awareness because the computers are doing it for them. It can go as far as forgetting to ever check on the system and its reliability.

Over Trust: The pilots start trusting the systems because of the fantastic job it does, and start no longer worry about the integrity of the systems and allow them to do the job. Some times they believe the computers over the other warning signs.

http://aviationknowledge.wikidot.com/aviation:glass-cockpits

The “Glass Cockpit” Problem

Over Confidence: With the ability to decrease the workload of the pilot, it means they can now complete more complex tasks during flight. However, this can create an illusion of "good piloting". The question that needs to be asked is "could you do this if the automation was off".

Reliance: The above problems often lead to reliance on the automation system. It is now being seen that pilot are losing their "flying skill" and its being replaced with supervising computer systems. The problem is some pilots can no longer fly the aircraft without the automated systems. (Skitka, 2000)

http://aviationknowledge.wikidot.com/aviation:glass-cockpits

Extrapolation to Bioanalytical Methods

Although focused on aviation, these principles can be applied to bioanalytical methods and data analysis where automation has begun to replace the human element in analysis.

An Observation on QualityQuality is neither necessarily expensive nor time

consumingLack of Quality is always Costly

◦ Financial, Cost to re-do work◦ Time, Delay to market◦ Reputational, Client loss of confidence in ability◦ Business, Loss of clients

Closing Thoughts

Would you stake your professional life right now on the quality of your

bioanalytical work?

If, not then WHY do you tolerate it?WHY do you think Your Professor/Client/

or a Regulatory Agency will?

“It is not enough to do your best; you must know what to do,

and then do your best.”W. Edwards Deming