A comparison of criteria for clinical validity and utility ...€¦ · CAPABILITY Subcommittee 1...

CAPABILITY Subcommittee 1 Draft ReportMay 2008

A comparison of criteria for clinical validityand utility in various national and

international frameworks

Prepared byJörg Schmidtke on behalf of Subcommittee 1Hannover Medical School, Institute of Human Genetics, Carl-Neuberg-Str. 1, 30625 Hannover, GermanySchmidtke.Joerg@mh-hannover.de

Subcommittee 1 Tasks

• systematic review of existing approaches of genetic test evaluation with an emphasis on clinical utility of tests with a potential to be implemented in general pract ice care and prevention

• included in the review will be genetic test evaluat ion approaches developed in the USA such as ACCE and EGAPP and the approaches of the United Kingdom Genetic Testing Network and the reviews done so far by Unit 3 of EuroGentest

• Draft of an analytic framework (=questions that nee d to be addressed) for evaluating genetic test applicati on in different settings

Chairperson will prepare the draft for the CAPABILI TY Working Group

Subcommittee 1 Tasks

CAPABILITY Subcommittee 1 MeetingFriday, 21st September 2007Sils Maria, Switzerland

Chair: Jörg Schmidtke

1. Report from the coordination office (Nippert)

2. Tasks of Subcommittee 1 (Nippert)

3. Frameworks developed so far and underlying definitions of clinical utility (joint discussion)• ACCE/problems applying ACCE criteria in practice • EGAPP Approach and topics under current review• CanGèneTest• Criteria developed by the UK GTN (Zimmern)• Criteria developed by EuroGentest Unit 3 (Schmidtke)• Disease-specific guidelines developed by the German Society of

Human Genetics (Schmidtke)• Paper for the United Kingdom Genetic Testing Network (Zimmern)

4. Frameworks developed so far and underlying definitions of clinical utility (joint discussion)

US Funded Projects:The ACCE Evaluation Process for Genetic Testing

AA nalytic validity

CC linical validity

CC linical utility

EE thical, Legal &Social implications

Analytic framework40 + targeted questions

Funded by CDC (2000-2004), Office of Genomics and Disease Prevention

What facilities/personnel are available or easily put in place?Facilities

What are the financial costs associated with testing?Health Risks

Is there general access to that remedy or action?Intervention

What health risks can be identified for follow-up testing and/orintervention?

Health Risks

What are the results of pilot trials?Pilot Trials

What quality assurance measures are in place?Quality Assurance

Is the test being offered to a socially vulnerable population?Intervention

Is there an effective remedy, acceptable action, or other measurablebenefit?

Intervention

If applicable, are diagnostic tests available?Intervention

What is the impact of a positive (or negative) test on patient care?Intervention

What is the natural history of the disorder?Intervention

Specific QuestionComponent

http://www.cdc.gov/genomics/gtesting/file/print/FBR/BCCliUti.pdf

ACCE - Clinical Utility

Specific QuestionComponent

What are the economic benefits associated with actions resulting from testing?

Economic

What guidelines have been developed for evaluating program performance?

Monitoring

Are there informed consent requirements?Education

What methods exist for long term monitoring?Monitoring

What educational materials have been developed and validated andwhich of these are available?

Education

ELSIWhat is known about stigmatization, discrimination, privacy/confidentiality and personal/family social issues?

Impediments

What safeguards have been described and are these safeguards in place and effective?

Safeguards

Are there legal issues regarding consent, ownership of data and/orsamples, patents, licensing, proprietary testing, obligation to disclose, or reporting requirements?

Impediments

ACCE - Clinical Utility

http://www.cdc.gov/genomics/gtesting/file/print/FBR/BCCliUti.pdf

US Funded Projects:Evaluation of Genomic Applications in Practice and

Prevention (EGAPP)

http://www.egappreviews.org/workingrp/topics_diagram.htm

Funded by CDC (2004-2008)

Topics Chart

*variants or mutations in the identified gene or genes

Completed Reports on: http://www.egappreviews.org/workingrp/reports.htm

EGAPP – Completed Topics (last update February 12, 2008)

Although research remains promising, adaptation of genomic tests into clinical practice must await

appropriately designed and powered studies in relevant clinical settings.

1) and 2) Detection and management

1) General population of

women and; 2) women at increased

risk for ovarian cancer

Genomic TestsOvarian Cancer

The sensitivity, specificity, and reliability of the tests used to evaluate individuals for suspected HNPCC is

not known confidently. Data regarding the net benefits and harms associated with predictive genetic testing in patients with HNPCC-related cancers and their families

members is incomplete

Management of individuals and early detection/prevention for

family members

Individuals diagnosed with CRC and their

family members

Mismatch repair gene mutations

Hereditary Nonpolyposis

Colorectal Cancer (HNPCC)

There is a paucity of good-quality data addressing the questions of whether testing for CYP450

polymorphisms in adults entering SSRI treatment for non-psychotic depression leads to improvement in outcomes, or whether testing results are useful in

medical, personal, or public health decision making.

Treatment with SSRI drugs

Individuals diagnosed with

depressionCYP450Depression

For all tests, the relationship of predicted to observed risk in different populations still needs further study, as

does their incremental contribution, optimal implementation, and relevance to patients on current

therapies.

Improve prognostic accuracy,

treatment choice and health outcomes

Women diagnosed with

early stage breast cancer

3 marketed gene expression based

assaysBreast Cancer

Intended UseTarget Population

Outcome Evaluation

Clinical Scenario

Test Assessed*Disorder/

Effect

Treatment with irinotecan

Individuals diagnosed �with CRC

UGT1A1Colorectal Cancer

(CRC)

Risk prediction or nutritional/lifestyle

management

General populationMultigene panel

Cardiovascular Disease

Intended UseTarget Population

Clinical ScenarioTest Being Assessed*Disorder/

Effect

*variants or mutations in the identified gene or genes

EGAPP – Topics under Review(last update February 12, 2008)

Completed Reports on: http://www.egappreviews.org/workingrp/reports.htm

The UKGTN has developed a “Gene Dossier“ process to evaluategenetic tests and recommend which

tests will be provided by theNational Health Service

UK Genetic Testing Network (UKGTN) Experience

Areas covered in the gene dossier include:

• The laboratory details of the test

• The test characteristics

• The clinical details of the condition

• The prevalence of the condition

• The purpose of the test

• The healthcare context in which the test is to be used

• The clinical utility of the test

UK Genetic Testing Network (UKGTN) Experience

Source: M. Kroese et al: How can genetic tests be evaluated for clinical use?, European Journal of Human Genetics (2007) 15, 917-921

The purposepurpose of a test must be specified in advance

because the effectivenesseffectiveness of the test is defined to

be the extent to which it fulfils the purpose for

which it was undertaken.

It is not possible to measure effectiveness without

defining purpose

Test Purpose

The purposes for which genetic testing are carried

out all fall within three categories. They are:

1. Reduce morbidity or mortality

2. Provide information salient to the care of the

patient or family members and/or

3. Assist the patient or family members with

reproductive decision-making

Test Purpose

• The time and effort to complete a genedossier was greater than originally predictedbecause of the gaps in genetic and laboratoryinformation

and

• the need to calculate test performancemeasures.

UK Genetic Testing Network (UKGTN) Experience

Source: M. Kroese et al: How can genetic tests be evaluated for clinical use?, European Journal of Human Genetics (2007) 15, 917-921

• This raised the question of the requisitebalance between the degree of detail requiredfor the evaluation of test performance

and

• the negative impact of not providing the test.

UK Genetic Testing Network (UKGTN) Experience

Source: M. Kroese et al: How can genetic tests be evaluated for clinical use?, European Journal of Human Genetics (2007) 15, 917-921

• The UKGTN experience of evaluatinggenetic tests for rare single gene disordershas shown clinical utility to be the mostimportant, and the most complex, of all theACCE domains.

UK Genetic Testing Network (UKGTN) Experience

Source: M. Kroese et al: How can genetic tests be evaluated for clinical use?, European Journal of Human Genetics (2007) 15, 917-921

• The UKGTN experience suggests that thiswas less of an issue for tests for highlypenetrant inherited diseases (where a modified evaluation would usually suffice) than for tests used in complex disorders(where complete and thorough evaluationwould most likely be required)

UK Genetic Testing Network (UKGTN) Experience

Source: M. Kroese et al: How can genetic tests be evaluated for clinical use?, European Journal of Human Genetics (2007) 15, 917-921

This discussion

framework was

produced for, and has

now been endorsed by

the UK Genetic Testing

Network

The PHG Foundation Framework for Genetic Test Evaluation

The PHG Foundation Framework for Genetic Test Evaluation (1)

Source: Burke, W., Zimmern, R.; Moving beyond ACCE: An Expanded Framework for Genetic Test Evaluation; PHG Foundation, 2007

The PHG Foundation Framework for Genetic Test Evaluation (2)

Source: Burke, W., Zimmern, R.; Moving beyond ACCE: An Expanded Framework for Genetic Test Evaluation; PHG Foundation, 2007

By the end of 2007, the EuroGenteststeering committee decided that the German

“indication criteria” are to be regarded as prototypic examples for a future set of

similar guidelines which will be developed by EuroGentest, endorsed by the European Society of Human Genetics, and published

in the European Journal of Human Genetics.

The EuroGentest Approach towards Genetic Test Evaluation

Aims at:

Developing criteria to determine when potential tests are ready to move from research to clinical practice

Involving leading experts from across Europe and elsewhere to develop evaluation tools

The EuroGentest Approach towards Genetic Test Evaluation

EuroGentest process for developingrecommendations for genetic test evaluation

EuroGentest Process, 1st Step

Data Collection

and analysis

Data Collection

and analysis

Expert led

Discussion

Expert led

Discussion

Draft

Recommendations

Draft

Recommendations

ConsultationConsultation

PublicationPublication

Survey 2006

Recommendations2008

http://content.karger.com/ProdukteDB/produkte.asp?A ktion=Ausgabe&Ausgabe=234201&ProduktNr=224224

� There is developing consenus on the four major domains of genetic test evaluation, namely analyticalvalidity, clinical validity, clinical utility, and et hical, legal and social implications (ACCE).

� While the ACCE definitions for analytical and clinicalvalidity are straightforward and globally applicable, criteria for clinical utility are highly context-dependent. Such contexts include the health caresystem within which a test is to be provided as well as locally available resources and set priorities. An evaluation of clinical utility is also stronglyinfluenced by surrounding ethical, legal, and socialissues.

Conclusions & Recommendationsbased upon the EuroGentest Survey

� The natural history of the disease, if known, should be con sideredso that testing and intervention can be properly timed.

� Interventions that might follow a positive test result sh ould beeffective and available.

� Qualified pre-test, test, and post-test measures, incl udingappropriate consent processes and genetic counseling, shou ld bein place when needed.

� Health risks associated with testing and interventions foll owingpositive and negative test results as well as with not testingshould be considered.

� Financial costs and benefits of testing should be evalua ted.� Testing services should provide educational materials, acce ss to

genetic counselling, and maintain surveillance over thei r activities.

Ideally, the following components should be considered whe n theclinical utility of a genetic test needs to be assessed :

Conclusions & Recommendationsbased upon the EuroGentest Survey

� Currently, these components can beassessed with sufficient confidence and reliability for a small number of conditionsonly, i. e. conditions with relatively high prevalence.

� For these, any genetic service system orprogramme should design disease-specific recommendations to guide test implementation in routine practice.

Conclusions & Recommendationsbased upon the EuroGentest Survey

� The great majority of conditions due to heritable mutations with high penetrance isso rare that sufficient quantitative data fordisease-specific guidelines are currentlynot available and may indeed never begathered.

� For assessing the clinical utility of genetictesting for such conditions, a muchabbreviated decision tree is proposed:

Conclusions & Recommendationsbased upon the EuroGentest Survey

1 assumed that the patient is clinically examined (including non-invasive intervention, e.g. imaging, elctrophysiology)2 Including predictive prenatal diagnosis 3 Predominantly processes with a high risk for patients – or other processes like biochemical tests instead of genetic tests

Decision Tree

� There is great heterogeneity in genetictesting services among the countriessurveyed.

� It is premature to mandate that genetictesting provided by clinical services meetsprofessional standards regarding clinicalvalidity and utility, because there is to date no consensus within the scientificcommunity and among health careproviders to what extent clinical validityand utility can and need to be assessed.

Conclusions & Recommendationsbased upon the EuroGentest Survey

Whenever limitations are necessary, healthcare systems may prioritize within genetictesting services according to prior risk, severity (if measurable in a meaningful way), disease prevalence, or availablitiy and cost of test methodology.

Conclusions & Recommendationsbased upon the EuroGentest Survey

• In Germany, the decision for or against prescribing/offe ringany particular genetic test is up to the individual docto r, in a case-by-case fashion

• No genetic screening programmes (except newbornscreening)

• One single specific guideline: management of ICSI –exclusion of CFTR mutation

• Prerequisites for a predictive test are fulfilled whenev er a fully informed patient considers it beneficial (BÄK guide lineof 2003)

• Medical measures are indicated provided they are"necessary, adequate and economic" (SGB V)

Disease-specific guidelines developed by the German Society of Human Genetics

The German Society of Human Genetics aims to move towards disease specific

guidelines rather than developing a framework for test evaluation. The

“indication criteria” are developed for some disorders based on ACCE model´s questions and are meant as a guide for

clinical practice.

Disease-specific guidelines developed by the German Society of Human Genetics

• Other relevant diagnosis methods• Economical aspects • Intervention • Prognosis • Management • Predictive setting• Genetic risk evaluation of the family members• Prenatal diagnosis • Other consequences

Clinical Utility

• Analytical Sensitivity• Analytical Specificity • Clinical Sensitivity • Clinical Specificity• PPV• NPV

Test characteristics

• Name and OMIM number of disorder and gene/chromosom e/segment• Spectrum of the mutations • Analysis method• Validation • Prevalence in Germany• Test application in which setting:

– (Differential) diagnosis– predictive diagnosis– risk evaluation of family members– prenatal diagnosis

Disorder/approach

Specific QuestionElement/Component

Disease-specific guidelines developed by the German Society of Human Genetics

List of diseases for which “Indication Criteria“ have be en completed(more than 25 diseases, as of May 2008 available in E nglish (www.eurogentest.org))

Poupak Javaher, MD, MPHFrauke Becker, Dipl. Oec.Markus Preylowski, Dipl. Oec.

Hannover Medical School

Ulf KristofferssonHelena KäriäiinenIrma NippertJorge Sequeiros

EUROGENTEST Unit 3 Partners

Acknowledgement