1
SCreening for Occult REnal Disease (SCORED)Simple Algorithms to Predict Kidney Disease: ready to be used in the real world?
Heejung Bang, PhD & Madhu Mazumdar, PhD Division of Biostatistics and Epidemiology
Department of Public Health
Weill Medical College of Cornell University
2
Overview
Background ObjectivesMethods: model development and
validation ResultsDiscussion
3
BackgroundPrevalence of Kidney Disease (1999-2004)
5.75.4 5.4
0.3 0.10
1
2
3
4
5
6
Stage 1 GFR>90Stage 2 90-60Stage 3 59-30Stage 4 29-15Stage 5 <15
Stages 1 and 2 with kidney damage
4
BackgroundEnd-Stage Renal Disease (ESRD) Counts
5
BackgroundTotal Cost of Medicare for ESRD (in billions)
28.3
14.2
0
6
12
18
24
30
1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010
95% CL
Projection
Cost
6
Background
Chronic kidney disease (CKD) is a global health problem. Low-awareness and late detection are common problems.
It is progressive disease. Yet, most affected individuals are asymptomatic with known risk factors and are not routinely tested.
Identifying individuals with CKD should be ‘simple’ with serum creatinine concentration that is widely available and inexpensive ($10-20), in combination with urinalysis.
Systematic methods to predict disease in other chronic conditions such as cardiovascular disease (e.g., Framingham, Reynolds scores, stroke instrument), cancer (e.g., Gail model), diabetes exist but not for CKD.
7
Objectives
To develop risk prediction model for prevalent CKD Important prerequisites in our investigation:
Easy to use but accurate Cumulative effects of concurrent risk factors Demographic + medical history + modifiable risk factors
To test the validity of the model internally as well as using independent large databases (i.e., external validation)
To compare the performance of the model with the current clinical practice guidelines
To develop risk prediction model for incident CKD
8
Kidney Early Evaluation Program (KEEP) by the National Kidney Foundation
if a persons is ≥ 18 years old and has one or more of the following:
1. diabetes
2. high blood pressure
3. a family history of diabetes, high blood pressure or kidney disease
http://www.kidney.org/news/keep/
9
SCreening for Occult REnal Disease
(SCORED)Bang et al. (2007)
10
Methods
Cross-sectional analysis of a nationally representative population based survey, the National Health and Nutritional Examination Surveys (NHANES) 1999-2002
Adult subjects only (≥20 years old) Potential risk factors searched from literature Endpoint: CKD stage 3 or higher, i.e., glomerular
filtration rate (GFR) < 60 ml/min/1.73m2 (using the MDRD formula)
11
Methods (Cont’d)
Split-sample method to create a development and validation dataset using a 2:1 ratio.
Standard diagnostic characteristics: # at high risk, sensitivity, specificity, positive & negative predictive values, area under ROC curve
Multiple logistic regression model (with proper weighting and complex survey design)
e.g., proc surveylogistic in SAS.
12
Methods (Cont’d)
‘Categorical scoring system’ derived by assigning an integer for the regression coefficients
‘Continuous probability’ of having CKD from the fitted regression model
External validation using the Atherosclerosis Risk in Communities (ARIC) Study, Cardiovascular Health Study (CHS) and NHANES 2003-2004.
Comparison between SCORED vs. KEEP using standard diagnostic measures
A number of sensitivity analyses (e.g., missing info, different definitions)
--- important to be used in the real world!
13
Results
NHANES 1999-2002 gave 10,291 individuals
After exclusions (based on unmeasured or missing data, etc.), dataset included 8,530 observations
A total of 601 individuals had CKD (5.4% weighted proportion)
14
Final SCORED model in development data (N= 5,666, AUC=0.88)
15
Diagnostic characteristics of SCORED in internal validation dataset (N=2,864) (cutpoint ≥4 to define high risk group)
16
Event rate by risk score
17
Fitting SCORED model to ARIC dataset (N= 12,038, AUC=0.71)
18
Sample questionnaire
19
Advantages of SCORED
Estimate the cumulative likelihood of having disease with multiple risk factors
Accuracy and high sensitivity. Simple to use (implemented by the pen & pencil
method) so foresee a variety of uses e.g., mass screenings public education initiatives, health fair
medical emergency departments web-based medical information sites patient waiting room in clinics
20
Limitations of SCORED
Inability to assess family history of kidney disease -- many large national and community studies do not
enquire about history of kidney disease. For prevalent disease, not incident disease (a new
risk score is needed, later in this talk) Some variables may be commonly missing (e.g.
proteinuria) Low PPV (but prediction is HARD!) Kidney disease: multiple definitions, different stages
21
Diagnostic performance of SCORED vs. KEEP using external validation data (Bang, Mazumdar et al. 2008)
Screening guidelines % high risk Sensitivity Specificity PPV NPV AUC
SCORED
NHANES 40 95 65 20 99 0.88
ARIC/CHS 51 88 52 1498
0.78
ARIC/CHS* 53 89 50 13 98 0.79
ARIC/CHS* 53 90 50 13 98 0.80
KEEP
NHANES 67 90 35 12 97 0.75
NHANES* 69 92 33 12 98 0.77
ARIC 76 88 24 3 98 0.67
ARIC/CHS 77 86 24 9 95 0.65
* some sensitivity analyses
22
A simple algorithm to predict incident kidney disease
(aka, SCORED II)
by Kshirsagar, Bang et al. In Press
23
Prediction is very hard, especially about the future - Yogi Berra
24
Background
Another important issue is to predict a new disease in disease-free population.
In many asymptomatic diseases, both prevalent and incident diseases are important. (in contrast, for hard outcomes such as heart attack, only incident disease makes sense)
Incident disease is less urgent so less user-friendliness is acceptable.
--- 3 different models developed: 1) best-fitting continuous, 2) best-fitting categorical, 3) simplified categorical.
Beyond AUC. We also used AIC/BIC.
25
Background (Conti’)
We need prospective studies to develop the models.
Internal validation only using Split-sample, no external validation.
Same logistic regression --- so observed outcome among survivors.
Cutpoint for high risk group might be less important.
26
Simplified categorical model (AUC=0.69, AIC=6295, BIC=6374)
CovariateBeta coefficient (standard error)
Odds Ratio (95% Cl)
P valueAssigned
score
Age 50-59 0.63 (0.12) 1.9 (1.5, 2.4) <0.0001 1
60-69 1.33 (0.12) 3.8 (3.0, 5.8) <0.0001 2
70 or older 1.46 (0.14) 4.3 (3.3, 5.6) <0.0001 3
Female 0.13 (0.07) 1.1 (1.0, 1.3) 0.05 1
Anemia 0.48 (0.20) 1.6 (1.1, 2.4) 0.02 1
Hypertension 0.55 (0.07) 1.7 (1.5, 2.0) <0.0001 1
Diabetes mellitus 0.33 (0.10) 1.4 (1.2, 1.7) 0.0006 1
History of cardiovascular disease
0.26 (0.10) 1.3 (1.1, 1.6) 0.009 1
History of heart failure 0.50 (0.25) 1.6 (1.0, 2.7) 0.04 1
Peripheral vascular disease 0.41 (0.13) 1.5 (1.2, 1.9) 0.002 1
27
Risk prediction table for up to 10 years
Total score Estimated Risk (%)
≤1 ≤5
2 8
3 13
4 20
5 25
6 30
7 35
≥8 ≥50
28
Discussion
Evidence-based medicine = Science (theory) + Data + Statistics.
Risk score = Statistics + Art + Reality
--- SCORED is a good example.☺ Performed well in a variety of different settings. Seems to provide the enhanced guidelines upon the current
clinical practice guidelines. It started be utilized in the ‘real world’. SCORED II yet to be validated but strong consistency/
similarities observed in SCORED I and II.
29
Discussion (Conti’)
Categorization can be a bad idea (Royston et al. 2005; Greenland 1995) but is crucial for risk scoring algorithms to be useful in the real world.
More than 1 model may be justified and we can let consumers/users to choose because
All models are wrong, but some are useful ---George Box Relying on only 1 measure (e.g., AUC) can be problematic
(Cook et al. 2006; Cook 2007). Trade-offs between accurate vs. easy medical terms. Risk scores for internet vs. physician’s office vs. Walmart can
be different.
30
Current and future research
Evaluation of SCORED in vascular patients because detection of CKD in patients with or at
increased risk of CVD was emphasized by a science advisory from the American Heart Association and National Kidney Foundation (2006).
Relationships SCORED with other risk scores Testing SCORED in community settings
31
References
BangBang, Vupputuri, Shoham et al. (2007). SCreening for Occult REnal Disease (SCORED). A simple prediction model for chronic kidney disease. Archives of Internal Medicine.
BangBang, Mazumdar, Kern et al. (2008). Validation and Comparison of a novel prediction rule for kidney disease: KEEPing SCORED. Arch Int Med.
Kshirsagar, BangBang, Bomback et al . A simple algorithm to predict incident kidney disease. In Press. Arch Int Med.
BangBang, Mazumdar, Newman et al. Screening for kidney disease in vascular patients. Submitted.
Building and Using Disease Prediction Models in the Real World. Roundtable discussion led by H. Bang at JSM, Utah, 2007. Slides at:
http://www.med.cornell.edu/public.health/conference_presentations.htm
32
Exposed to and used by public
Covered by the CBS Early Show (on World Kidney Day 2007)
SCORED questionnaire is posted in some health information websites
Distributed by ESRD network, KidneyTrust, Am Kidney Fund, UK Dept of Health, and UNC Kidney Center for Kidney Education Outreach Program
“Research Highlights” in Nature Clinical Practice Nephrology (2007)
Lead Story in Physician’s Weekly (2007)