Short and Precise Patient Self-Assessment of Heart FailureSymptoms Using a Computerized Adaptive Test

Matthias Rose, MD, PhD; Milena Anatchkova, PhD; Jason Fletcher, PhD; Arthur E. Blank, PhD;Jakob Bjørner, MD, PhD; Bernd Lowe, MD, PhD; Thomas S. Rector, PhD; John E. Ware, PhD

Background—Assessment of dyspnea, fatigue, and physical disability is fundamental to the monitoring of patients withheart failure (HF). A plethora of patient-reported measures exist, but most are too burdensome or imprecise to be usefulin clinical practice. New techniques used for computer adaptive tests (CATs) may be able to address these problems.The purpose of this study was to build a CAT for patients with HF.

Methods and Results—Item banks of 74 queries (“items”) were developed to assess self-reported physical disability,fatigue, and dyspnea. All queries were administered to 658 adults with HF to build 3 item banks. The resulting HF-CATwas administered to 100 patients with ancillary HF (New York Heart Association I, 11%; II, 53%; III and IV, 36%).In addition, the physical function and vitality domains of the SF-36 Health Survey questionnaire, an establishedshortness-of-breath scale, and the Minnesota Living with Heart Failure Questionnaire were applied. The HF-CATassessment took 3:09�1:52 minutes to complete and score. All HF-CAT scales demonstrated good construct validitythrough high correlations with the corresponding SF-36 Health Survey physical function (r��0.87), vitality(r��0.85), and shortness-of-breath (r�0.84) scales. Simulation studies showed a more precise measurement of allHF-CAT scales over a larger range than comparable static tools. The HF-CAT scales identified significant differencesbetween patients classified by the New York Heart Association symptom criteria, similar to the Minnesota Living withHeart Failure Questionnaire.

Conclusions—A new CAT for patients with HF was built using modern psychometric methods. Initial results demonstrateits potential to increase the feasibility and precision of patient self-assessments of symptoms of HF with minimizedrespondent burden.

Clinical Trial Registration—URL: http://www.projectreporter.nih.gov. Unique identifier: 1R43HL083622-01.(Circ Heart Fail. 2012;5:331-339.)

Key Words: heart failure � patient-reported outcomes � computer adaptive tests

The cardinal manifestations of heart failure (HF) aredyspnea and fatigue, limited tolerance of physical activ-

ity, fluid retention, pulmonary congestion, and peripheraledema. Therefore, HF is a clinical diagnosis that is largelybased on physical examination and a careful history abouttypical subjective symptoms in the presence of cardiacdysfunction.1 A patient-centered measurement approach isparticularly important in HF, to provide clinicians with toolsto help them to monitor the syndrome, to compare improve-ments under different forms of therapy, and to identify risk ofdeterioration. The New York Heart Association (NYHA)classification has been used for this purpose, but it is beingcriticized for its questionable reliability2,3 and is rarely usedoutside clinical studies or specialized units.

Clinical Perspective on p 339Generally, patient self-assessments have been the more reli-

able assessments of subjective symptoms, which is one reasonfor a growing interest in subjective health status measures fromthe scientific community, clinical practitioners, and industry.4,5

Self-assessed symptoms are used to predict declines in healthstatus of patients with HF,6 total expenses for HF care,7 hospi-talization, or even mortality.8,9 Their widespread use has beenrecommended to increase quality of care,10 and 30% of all newdrug developments use patient-reported outcomes (PROs) astheir primary or coprimary end point.11

However, with traditional methods, a comprehensive andreliable “static” measure is likely to be long and time-consuming to administer and score. If questionnaire data need

Received June 28, 2011; accepted March 20, 2012.From the Department of Quantitative Health Sciences, University of Massachusetts, Worcester, MA (M.R., M.A., J.E.W.); the Department of

Psychosomatic Medicine, Charite–University Medicine, Berlin, Germany (M.R.); Department of Psychosomatic Medicine, University Medical CenterHamburg-Eppendorf and Schon Klinik Hamburg-Eilbek, Hamburg, Germany (M.R., B.L.); the Department of Family and Social Medicine, AlbertEinstein College of Medicine, Bronx, NY (J.F., A.E.B.); 3i QualityMetric, Lincoln, RI (J.B.); Veterans Affairs Medical Center and Department ofMedicine, University of Minnesota, Minneapolis, MN (T.S.R.); and John Ware Research Group, Incorporated, Worcester, MA (J.E.W.).

Correspondence to Matthias Rose, Department of Psychosomatic Medicine, Charite–University Medicine Berlin, Chariteplatz 1, 10117 Berlin,Germany. E-mail matthias.rose@umassmed.edu

© 2012 American Heart Association, Inc.

Circ Heart Fail is available at http://circheartfailure.ahajournals.org DOI: 10.1161/CIRCHEARTFAILURE.111.964916

331

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

to be analyzed manually, assessments become cost prohibi-tive for use in routine clinical practice, and individual patientreports cannot be provided in a timely manner. Short formslimit the respondent burden but often show more ceiling orfloor effects and lack the precision required at the individualpatient level.12,13 Measurement precision to guide individualdecision making must be substantially higher than for groupcomparisons, because true change must be separated frommeasurement error for every single assessment.13 For exam-ple, if a CI of 95% is required, a traditional tool with goodpsychometric properties for group comparisons (eg, Cron-bach ��0.80) would only allow for interpretation of scoredifferences of almost 1 SD when used for an individual.14

Moreover, classic psychometric methods cannot be used todetermine the measurement precision for an individual mea-surement. As a result, none of the existing tools has becomea standard measure in clinical practice.15,16 Enhancing theprecision, accessibility, and interpretability of PRO measurescould make HF management more efficient and effective inmeeting patient care needs.

With the presented study, we apply computerized adaptivetesting (CAT) methods, a measurement technology17 that isused widely in educational testing.18 We aimed to build asystem that will allow routine, comprehensive assessment ofpathognomonic symptoms. The use of CAT techniques alsopromises to provide more precise measures, with fewer items,and an effective resolution to the classic conflict betweenpracticality and precision faced by traditional measurementmethods.12 The CATs tailor each assessment to the individ-ual’s status on what is being measured, applying only itemsthat are most appropriate for her or his current health status.Responses to each CAT item direct the choice of thefollowing CAT item toward the most informative for thisparticular assessment. A patient indicating higher levels ofdisability within the first questions would only be asked aboutthis level of ability. Omitting the use of uninformative items notrelevant for a given functional limitation focuses the assessment,decreases the respondent burden, and increases the measurementprecision achievable with a given number of items.

The CATs select the items out of a larger item bankrepresenting the entire range of the construct being measured.Most of the item banks are built on the principles of theItem-Response Theory (IRT). The National Institutes ofHealth are intensively promoting the use of these methods todevelop a comprehensive Patient-Reported Outcomes Mea-surement Information System (PROMIS) as part of theirroadmap initiatives (http://nihroadmap.nih.gov/). The authorsof this article are part of the PROMIS initiative, which aimsto provide a standard assessment for generic health statusmeasures in the near future.19

The goal of this study was to develop CATs for dyspnea,fatigue, and physical function for the assessment of patients withHF and to evaluate their acceptability, precision, and validity.

MethodsDevelopment of the ItemsAfter review of the relevant literature, we developed a set of 74patient questions (items) covering the 3 primary physical impair-ments commonly reported by patients with HF: physical function/

disability (24 items), dyspnea (30 items), and vitality/fatigue (20items). The queries were designed to be short enough to fit on aportable telephone screen for home assessments (Figure 1). Itemswere selected to represent the entire continuum of each aspect of HFfrom no to severe impairment. All 3 item banks have been scored inthe direction that higher scores indicate more impairment (ie,physical disability, fatigue, and dyspnea).

The item bank development was performed separately for eachof the 3 domains of physical function, dyspnea, and fatigue,following the same procedures as described in previous studies.20,21

After the item banks were developed, we used them as a basis for aCAT. A new software solution was developed to work on a PersonalDigital Assistant. The CAT logic can be set to stop after themeasurement reaches a particular precision or after a maximum ofitems is administered. For this study phase, the CAT was set to assesseach of the 3 different domains with an SE �3.3 (corresponding toa reliability of Cronbach � �0.90 for samples with an SD of 10) ora maximum of 7 items per scale.

ParticipantsThe data for the CAT item bank development sample were collectedvia the Internet from English-speaking adults with HF. All respon-dents were recruited by YouGov. YouGov uses a method calledsample matching for the selection of study samples from pools ofopt-in respondents.22 Sample matching starts with an enumeration ofthe target population. For patient recruitments, the target populationis all adults with similar sociodemographic characteristics, such aspatients with a particular condition, as enumerated in consumerdatabases (eg, maintained by Acxiom, Experian, and InfoUSA).Then, a random sample is drawn from the target population. Finally,

Figure 1. The heart failure (HF)–computer adaptive test (CAT)patient interface and examples for 1 item of each bank.

332 Circ Heart Fail May 2012

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

for each member of the target sample, a matching member of theInternet pool of opt-in respondents is selected, resulting in a“matched sample.” Matching was based on age, sex, and race. Theresulting matched sample has similar characteristics to the targetpopulation and will have similar properties to a true random sample.For this study, 14 028 adults were approached until the target numberof patients with HF was enrolled. All newly developed items wereadministered randomly.

The same data collection method and vendor were used for manysimilar projects, including an National Institutes of Health roadmapinitiative for the development of generic PRO tools (http://www.nih-promis.org). To ensure a sufficient distribution of responses for theitem parameter estimation, we used a quota of one third of patientswith minor, medium, and severe impairment based on 1 screeningquestion describing the level of impairment analogous to the NYHAclassification (I, II, and �III).

To help ensure the quality of the data, we applied the followingexclusion criteria: (a) average answering time per item was �5seconds, (b) subjects who did not indicate they had HF and 1underlying cause for HF, (c) subjects who did not indicate that theHF diagnosis was given by a physician, (d) last visit to a physicianwas �6 months ago, or (e) current medication did not indicate atleast 1 drug used for the treatment of HF (diuretics, angiotensin-converting enzyme inhibitor or angiotensin II receptor blocker,�-blockers, or digoxin).

To examine the characteristics of the HF-CAT, different simula-tion studies were conducted, as previously described.20,23 Theseanalyses are based on the real data provided for all items in the bankby the patients in the online survey. Only small subsets of those itemresponses are used to estimate the patient score for the CATsimulation (in IRT terms, called “� score”). The quality of the itemsin the bank defines the precision of the score at different ranges. The“test information curve” identifies floor and ceiling effects and if themeasurement range of the tool fits to the symptoms of the sample. Toillustrate this for the HF-CAT, the precision of the score estimatewas plotted as a function of the patient scores.20

To evaluate the construct validity of the HF-CAT, items from thefollowing established tools were also included in the data collection:the SF-36 Health Survey scales for physical functioning and vital-ity,24 4 items from the Medical Health Outcomes Survey to assessshortness of breath,25 and the Minnesota Living with Heart FailureQuestionnaire26 (MLHFQ, 21 items) as a legacy tool for measuring HFas indicated by patients’ perceptions of its overall effects on their lives.

A separate sample of 100 consecutive participants was recruitedfor the validity test conducted at the HF clinic of the MontefioreMedical Center, Bronx, NY. The clinic was selected because itusually does not use PRO assessments and predominantly serves alow-income diverse population. We considered this environment asparticularly challenging to test a new technology, assuming rela-tively low health literacy levels. In addition, we believed that anevaluation of psychometric properties would be more relevant in aless educated sample, because the validity of the IRT assumptionshas been already evaluated in the developmental sample, which wasaffluent and well educated (Table 1). Patients with previouslydiagnosed HF were invited to participate in the study. Consentingparticipants were asked to complete the actual HF-CAT on ahand-held computer (personal digital assistant [PDA]) and a series ofpaper-and-pencil assessments, including sociodemographic ques-tions, the MLHFQ, and a survey evaluation of the experience withthe HF-CAT. All participants completed both instruments. Partici-pants were randomly assigned to 1 of 2 groups within a crossoverdesign, in which the order of presentation of the HF-CAT assessmentand the MLHFQ was counterbalanced. Patients were placed in thewaiting area and asked to follow the standard instructions providedfor each measure.

Medical information, including NYHA class, was extracted fromthe medical files. The NYHA class is determined routinely for allpatients at every visit at the Montefiore Medical Center Heart FailureClinic based on the clinical assessment of the treating physician. TheNYHA class was determined without knowledge of the results of

patient self-assessments. Patients gave written informed consent andreceived a $25 incentive for their participation in the study.

ResultsSamplesAfter applying the inclusion and exclusion criteria, the finalitem bank development sample consisted of 658 participants,aged 60�13 years (49% female), who had experienced HFfor 8.8�7.9 years (Table 1). Patients reported the followingconditions in addition to their HF: 43%, coronary heartdisease; 42%, previous heart attacks; 18%, cardiomyopathy;14%, valvular heart disease; 5.2%, rheumatic fever; 60%,

Table 1. Characteristics of the Samples

Characteristics

HF-CAT

Development(IB Sample)

(N�658)

Evaluation(MMC Sample)

(N�100)

Age, y* 60 (13) 58 (12)

Time with HF, y* 8.8 (7.9) 4.6 (4.5)

Family status

Living in partnership 78 54

Living alone 21 33

Female sex 49 38

Hispanic or Latino ethnicity 4 35

Race

White 93 19

African American 3 46

Other 4 35

Education

�8th grade 0.1 13

Some high school 3 21

High school graduate 15 25

Some college 39 24

College graduate 22 11

Postgraduate 20 5

Household income, $

�5000 1 11

5001–20 000 18 22

20 001–45 000 32 15

45 001–75 000 23 10

�75 000 17 5

Prefer not to answer 9 37

Employment status

Student 0.3 4

Working at a paying job 22 23

Retired 56 47

Laid off or unemployed 3 2

A full-time homemaker 7 9

Other 11 11

Data are given as percentage of each group unless otherwise indicated. HFindicates heart failure; CAT, computer adaptive test; IB, item bank; MMC,Montefiore Medical Center.

*Data are given as mean (SD).

Rose et al Heart Failure CAT 333

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

hypertension; 31%, arrhythmias; and 40%, diabetes. Alcoholabuse was reported by 5.9% of patients.

The Montefiore Medical Center clinical sample (n�100)was predominantly male (62%), with a mean age of 58 years.The sample was diverse, including mostly African Americanpatients and many Hispanics. One third of the population hada comparatively low household income. The severity of theirHF symptoms, assessed by the NYHA classification, was11% in class I, 53% in class II, and 36% in class III or IV.

HF-CAT Development

Item Banks DevelopmentIn the final calibrated item banks, there were 21 itemsassessing physical disability, 20 items assessing fatigue, and29 items in the dyspnea bank with satisfactory item fit (Table2). Most informative (ie, with a high discrimination parame-ter: “slope”) was the item asking about the ability to runerrands, an item referring to a feeling of being “worn out,”and the item asking if the patient will be short of breathwalking from one room to another.

Simulation StudiesThe precision of every score estimate can be displayed as afunction of the level of function or the severity of thesymptoms. The results of the simulation studies showed thata highly precise score (comparable to an internal consistencyof ��0.90) can be estimated with 5 items for each domainover a range of nearly 3 SDs (Figure 2, left).

The concordance between the results of the CATs and theentire item bank was good for all of the constructs, asillustrated by the extremely high correlations (r�0.95–0.97),showing that the 5-item CAT can essentially capture theinformation provided by the entire bank. As expected, therewere high correlations between the simulated CAT scalescores and the corresponding SF-36 Health Survey’s physicalfunction (r�–0.87) and vitality (r�–0.84) scales, as well asthe static shortness of breath measurement (r�0.83). Com-pared with all legacy tools, the HF-CAT provides a moreprecise measurement over a larger measurement range (Fig-ure 2, right). For physical disability, a similar measurementprecision, such as with the SF-36 physical function scale, canbe achieved with 1⁄2 the number of items (Figure 2, top left).

HF-CAT Evaluation

Respondent BurdenOn average, 4 to 5 items were administered for the assess-ment of physical disability, fatigue, and dyspnea to achievethe predefined level of precision (Table 3). The average timefor administration of the entire HF-CAT with all 3 domainswas 3 minutes (3�2 minutes).

ValidityWe used the MLHFQ to help evaluate the constructs of theHF-CAT and the NYHA class to evaluate its discriminativevalidity (Table 3). The mean MLHFQ score of the samplewas 38�25, and the mean scores of the HF-CAT were59.6�8.4 for physical disability, 52.6�8.5 for fatigue, and54.8�13.3 for dyspnea. There were no order effects for anymeasure. The HF-CAT scales for physical disability, fatigue,

and dyspnea correlated significantly with the MLHFQ totalscore (r�0.71, r�0.63, and r�0.68, respectively).

A general linear model was used to evaluate the ability ofthe HF-CAT scales to statistically differentiate patients withdifferent levels of symptom severity, as measured by theclinician’s NYHA classification (Table 3). The main effectsfor all the measures were significant, with similar discrimi-native ability (Eta2, F values) for the HF-CAT physicaldisability and dyspnea scales and the MLHFQ scale.

User ExperienceBecause this study took place in a low-income, less educated,minority population, we were particularly interested in thesubjective user experience with a computer assessment. Ofthe patients, 98% found the HF-CAT assessment overall veryeasy or easy, 100% thought it was very easy or easy to followthe instructions, and 95% said it was very easy or easy to readthe questions on the screen. In addition, of the patients, 98%judged the time for the assessment as “just right,” and 90%considered the questions as relevant; 98% were willing to usethe device again on the next visit.

DiscussionFor the first time, to our knowledge, we applied computerizedadaptive testing methods to develop and evaluate an ultra-short assessment system for patients with HF (HF-CAT) inclinical practice. The tool allows routine, comprehensiveassessment of 3 primary problems that are commonly expe-rienced by patients with HF. If the emotional or social impactof the disease is of additional interest, further tools (eg, fromthe PROMIS) need to be added for a comprehensive coverageof the health-related quality-of-life construct.

FeasibilityThe feasibility of the HF-CAT in its PDA version wasevaluated in a low-income, low-educated, minority popula-tion in the Bronx. The HF-CAT is a practical and well-accepted tool. Nevertheless, it was tested under study condi-tions, and participants might have been biased receiving anincentive for their participation. To our knowledge, only 1report about the acceptance of CATs within clinical practicesettings is available. A similar CAT, also being displayed ona PDA, has been in routine clinical use since 2004. Patientsanswering this CAT also report a high acceptability. Almostall of the 423 consecutive patients considered the handling aseasy and believed that the use of the PDA made sense.27

Several other studies report about the reception of CATsunder study conditions. Most patients in a feasibility test of apain CAT found the CAT application to be useful, relevant,of appropriate length, and easy to complete.28 Similarly, mostrespondents in a feasibility study of an asthma impact CATfound it easy to complete and of appropriate length.29 Theresults of a feasibility test of a diabetes CAT gave somewhatmixed results. Although both English- and Spanish-speakingparticipants agreed that a paper-and-pencil assessment wasmore burdensome than a CAT, the Spanish-speaking partic-ipants preferred the paper tool and were more willing tocomplete a paper tool in the future.30

334 Circ Heart Fail May 2012

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

Table 2. IRT Item Parameters: HF-CAT Item Banks

Parameter Slope

Thresholds

Mean 1 2 3 4

Physical disability

Exercising hard for 0.5 h (1) 2.549 0.556 �0.123 1.236

Doing 1 h of physical labor (1) 2.810 0.625 0.072 1.177

Walking up a steep hill (1) 3.558 0.748 �0.154 1.650

Rearranging furniture at home (1) 3.952 1.136 0.610 1.663

Doing chores (1) 4.252 1.391 0.765 2.017

Doing daily physical activities (2) 3.432 1.492 0.537 1.191 1.715 2.523

Climbing up a flight of stairs (1) 3.728 1.535 0.824 2.246

Doing daily physical activities (1) 4.092 1.583 0.881 2.285

Carrying 2 bags of groceries (1) 3.800 1.595 1.132 2.058

Walking on flat ground (1) 5.247 1.621 1.621 *

Preparing a meal (1) 5.554 1.643 1.643 *

Walking 100 yards (1) 3.977 1.648 1.158 2.137

Standing up from a chair (1) 3.837 1.814 1.814 *

Running errands and shopping (1) 5.713 1.826 1.236 2.417

Dressing myself (1) 4.825 1.832 1.832 *

Taking a tub bath (1) 2.683 1.869 1.601 2.137

Getting from one room to another (1) 5.494 1.907 1.907 *

Standing up from a bed (1) 3.922 1.910 1.910 *

Getting on and off the toilet (1) 3.890 1.953 1.953 *

Making the bed (1) 4.330 1.955 1.444 2.465

Putting a trash bag outside (1) 4.768 1.995 1.536 2.453

Fatigue

Full of energy (3) 2.419 �0.475 �1.407 0.456

Strong and vital (3) 2.243 �0.421 �1.271 0.429

Fresh and rested (3) 1.979 �0.175 �1.195 0.845

Lively (3) 1.925 �0.131 �1.100 0.839

Active (3) 1.856 �0.123 �1.203 0.957

Full of life (3) 1.600 0.063 �0.756 0.881

Tired (3) 2.591 0.406 �0.638 1.450

Fatigued (3) 3.617 0.546 �0.345 1.436

Sluggish (3) 2.899 0.578 �0.383 1.539

Worn out (3) 4.090 0.647 �0.214 1.508

Run down (3) 3.445 0.679 �0.192 1.551

Wide awake (3) 1.217 0.741 �0.407 1.889

As if I have no energy left (3) 3.189 0.767 �0.072 1.606

Spent (3) 3.325 0.807 �0.104 1.719

Exhausted (3) 3.392 0.811 �0.016 1.637

Weary (3) 2.614 0.852 �0.042 1.747

Weak (3) 2.421 0.866 �0.094 1.825

Save my energy (3) 1.161 1.065 �0.064 2.195

Sleepy all day (3) 1.765 1.125 0.139 2.111

Jaded (3) 1.241 1.809 0.817 2.801

Dyspnea

Running a short distance makes me short of breath (3) 1.190 �0.525 �2.072 �0.090 0.587

Exercising hard for 0.5 h makes me short of breath 1.134 0.131 �0.206 0.468

Talking while walking up a hill will make me short of breath (3) 2.040 0.185 �1.455 0.385 1.625

An hour of physical labor makes me short of breath (4) 1.418 0.394 0.033 0.755

My breathing problems limit my ability to exercise as much as I would like (3) 1.500 0.449 �0.529 0.685 1.193

(Continued)

Rose et al Heart Failure CAT 335

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

Respondent BurdenOne important contribution of the CAT technology will be toreduce the respondent burden without compromising theprecision and validity of the assessment, by tailoring eachassessment to the patient’s condition. This advantage waspreviously demonstrated in a simulation study of the Activ-ities of Daily Living CAT, which found that the CATprovided similar results to a static version while reducing thenumber of items administered by 50%.31 Results from otherstudies indicate that scores similar to those obtained withfull-length item banks (ranging in length from 18–585 items)can be achieved through much shorter CATs when measuringfunctional status,32–34 mental health status,21,27,35,36 or theimpact of conditions, such as headache,23,37 diabetes,30

chronic pain,28 and asthma.29 Most actual CAT applications

used between 5 and 7 items to measure 1 construct. Thepresent HF-CAT applied between 4 and 5 items per scale, andthe average total time for the entire assessment and scoringwas 3 minutes (ie, 1 minute per scale, which could be appliedindividually). The assessment time of the MLHFQ elec-tronically measured in a previous study was 4�2 min-utes,38 and time to administer the Kansas City Cardiomy-opathy Questionnaire, another common tool for theassessment of patients with HF, is reported to be 4 to 6minutes without scoring.39

In summary, the HF-CAT provides a precise measure overa large measurement range with minimal respondent burden.As far as it is known today, it seems that CATs offer aneffective resolution to the classic conflict between practicalityand precision faced by traditional measurement technology.12

Table 2. Continued

Parameter Slope

Thresholds

Mean 1 2 3 4

Talking while walking up a flight of stairs makes me short of breath (3) 2.068 0.564 �0.919 0.807 1.803

During a typical day, I feel short of breath (4) 2.407 0.649 �0.220 1.518

Doing chores, such as vacuuming or yard work, makes me short of breath (3) 2.033 0.693 �0.608 0.763 1.924

Climbing up 1 flight of stairs makes me short of breath (3) 2.440 0.819 �0.646 0.983 2.120

Going outside for a walk makes me short of breath (3) 2.646 1.037 �0.213 1.197 2.128

Walking 100 yards makes me short of breath (3) 2.351 1.052 �0.064 1.194 2.027

Walking up a hill makes me short of breath (4) 2.122 1.085 0.381 1.789

Carrying groceries makes me short of breath (3) 2.796 1.102 �0.106 1.222 2.190

Talking while walking makes me short of breath (3) 2.422 1.241 �0.023 1.313 2.434

Running errands makes me short of breath (3) 2.677 1.351 0.191 1.415 2.448

Taking a bath makes me short of breath (4) 2.849 1.404 1.009 1.800

Dressing myself makes me short of breath (4) 3.118 1.431 0.887 1.975

Preparing a meal makes me short of breath (4) 3.104 1.451 0.988 1.914

Singing or humming makes me short of breath (4) 2.086 1.456 0.943 1.970

Speaking in a group makes me short of breath (4) 1.900 1.481 0.994 1.969

I feel short of breath when I sit and rest (4) 2.775 1.543 1.543 *

Talking at noisy places makes me short of breath (4) 2.187 1.606 1.170 2.043

Walking from one room to another makes me short of breath (4) 3.909 1.647 1.154 2.139

Talking to someone makes me short of breath (4) 2.875 1.779 1.247 2.311

Talking on the telephone makes me short of breath (4) 2.768 1.840 1.398 2.281

Getting off the bed makes me short of breath (4) 2.958 1.849 1.305 2.393

Going to the toilet makes me short of breath (4) 2.868 1.900 1.501 2.298

Lying down flat makes me short of breath (3) 1.532 1.924 1.234 2.000 2.537

Standing up from a chair makes me short of breath (4) 2.511 1.924 1.302 2.547

The table is ordered by the mean threshold value. Response options are as follows: 1, easy/hard/impossible; 2, no difficulty/a little bit of difficulty/some difficulty/alot of difficulty/cannot do because of my health; 3, not at all/somewhat/very much; 4, not at all/a little bit/quite a lot/cannot do; and 5, not at all/a little bit/quite alot. The IRT item bank parameters are developed as usual on a 0�1 metric, with 0 representing the scaling sample mean with an SD of 1. For easier interpretability,estimated patient scores are transformed linear to a 50�10 metric later. The slope parameter is also called the discrimination parameter. Higher slope parametersindicate a better discrimination, which makes the item more valuable (ie, “informative” for the score estimation: eg, the capability to “run errands” is more informativeto determine the physical disability of a patient than her or his ability to “put the trash outside the house”). The thresholds of an item show at which score level aparticular response option is the most likely to be endorsed. For the item “running errands,” the threshold 1.236 separates the response “easy” from “hard” andthe threshold 2.417 separates the response “hard” from “impossible.” If a patient scores 3 SDs higher than the population mean, she or he is most likely to answer the item“running errands is …” with “impossible” because her or his score is higher than the threshold of 2.417. If her or his level of disability is only 1.5 SDs higher than the USpopulation mean, she or he is likely to endorse “hard” because the score is between the thresholds 1.236 and 2.417. The mean threshold illustrates the position of the itemon the metric, which can be seen as “item difficulty” in traditional terms. The table is sorted by the mean threshold. IRT indicates Item-Response Theory; HF, heart failure;CAT, computer adaptive test.

*The 2 highest response options were collapsed for the item parameter estimation, the presentation of response options for the patient remains the same.

336 Circ Heart Fail May 2012

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

ValidityStudies of CAT applications in diseases, such as depres-sion27,35 or headache,40 have shown that their measurementadvantages can transfer to increased validity in identifyingdifferences between groups known to differ in clinical char-acteristics, compared with static tools. The 3 scales of theHF-CAT also discriminated between groups of patients ofdifferent NYHA classification equally and a legacy toolmeasuring the impact of HF, using 4 times more items. Theseinitial results show that the HF-CAT has the potential toprovide a valid, highly relevant assessment of patientswith HF.

Serial MeasurementsFor the assessment of patients with HF, we believe it isimportant to assess the health status of the patient at the point

of care and at the patient’s home. Because many elderlypatients do not have access to the Internet or are not familiarwith its use, one way to do so is the use of a “smart phone”and/or interactive voice recognition. Most established toolsinclude items that are not suitable to be used over thetelephone. The IRT methods allow using much simpler itemsover the telephone and more comprehensive items at thephysician’s office, and scoring both assessments on the samemeasurement metric. This allows having a smart phoneadminister the HF-CAT at the patient’s home and having thesame patient answering the more comprehensive PROMIS-CAT on a tablet PC at the physician’s office. The IRT-basedmeasurements of health outcomes are independent of theparticular items being administered and of the test adminis-trator. The same value for the same domain yields the sameinterpretation, whereas results from different traditional tools

Figure 2. Measurement precision in relation to measurement range. The x axis shows the patient score. In Item-Response Theory (IRT)terminology, this score is referred to as the “� score.” To make the heart failure (HF)–computer adaptive test (CAT) and the legacy toolscomparable, both instruments are scored on the same metric, as determined by the developed item banks. The y axis shows the 95%CI of the patient score; the smaller the y value, the higher the precision of the score. The dotted lines show CIs that would be compa-rable to an internal constancy of Cronbach � 0.80, 0.90, and 0.95, for illustrative purposes.

Rose et al Heart Failure CAT 337

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

cannot be compared directly, making serial health statusmonitoring less practicable.

LimitationsDespite many encouraging findings with recent CAT devel-opments, several issues still need to be addressed. Within thisstudy, we have only used outpatients to evaluate the HF-CAT, which limits the generalizability to less severelydisabled patients. However, one of the most relevant advan-tages of CATs is that they can essentially eliminate floor andceiling effects by applying items tailored to the test taker. Oursimulation studies have shown that the current item bankcovers �3 SDs above the population mean, which is where ahospitalized population of patients with HF usually scores.

We did not evaluate the test-retest reliability for theHF-CAT. Similarly, we have not used the HF-CAT in anintervention study to test its responsiveness to treatments.However, several studies have reported on the ability of otherCATs to detect change. For example, in a telephone study of540 patients with headache, a CAT for headache impact wasmore responsive to self-evaluated changes of headache im-pact than a corresponding 54-item bank.23 In a longitudinal,prospective cohort study of 94 patients discharged frominpatient rehabilitation, the CAT version of the ActivityMeasure for Post-Acute Care was comparable in responsive-ness to the 66-item static version.41 Similarly, in a series ofarticles, Hart et al33,34 report on the results of validationstudies of condition-specific CATs, using large data sets frompatients receiving rehabilitation services across multiple USclinics.

SummaryIn summary, we have developed a promising method tomeasure patient-reported dyspnea, fatigue, and physical func-tion for use in the care of patients with HF. This new measureis part of a rapidly growing number of new assessment toolsusing the advantages of item response theory and computer-ized adaptive test techniques,16,19,42 with some of them beingused in clinical practice already.27,43 However, whether theseencouraging improvements in measurement will transfer toimproved care and ultimately health of patients with HFwarrants further studies.

Sources of FundingThis study was supported in part by grant 1 R43 HL083622-01 fromthe National Institutes of Health/National Heart, Lung, and BloodInstitute (Dr Rose).

DisclosuresThe HF-CAT software was developed by QualityMetric Inc. Dr. J.Bjørner is an employee of this company.

References1. Hunt SA, Baker DW, Chin MH, Cinquegrani MP, Feldman AM, Francis

GS, Ganiats TG, Goldstein S, Gregoratos G, Jessup ML, Noble RJ,Packer M, Silver MA, Stevenson LW, Gibbons RJ, Antman EM, AlpertJS, Faxon DP, Fuster V, Jacobs AK, Hiratzka LF, Russell RO, Smith SCJr. ACC/AHA guidelines for the evaluation and management of chronicheart failure in the adult: executive summary: a report of the AmericanCollege of Cardiology/American Heart Association Task Force onPractice Guidelines. J Am Coll Cardiol. 2001;38:2101–2113.

2. Bennett JA, Riegel B, Bittner V, Nichols J. Validity and reliability of theNYHA classes for measuring research outcomes in patients with cardiacdisease. Heart Lung. 2002;31:262–270.

3. Goldman L, Hashimoto B, Cook EF, Loscalzo A. Comparative reproduc-ibility and validity of systems for assessing cardiovascular functionalclass: advantages of a new specific activity scale. Circulation. 1981;64:1227–1234.

4. Lett HS, Blumenthal JA, Babyak MA, Sherwood A, Strauman T, RobinsC, Newman MF. Depression as a risk factor for coronary artery disease:evidence, mechanisms, and treatment. Psychosom Med. 2004;66:305–315.

5. Konstam V, Moser DK, De Jong MJ. Depression and anxiety in heartfailure. J Card Fail. 2005;11:455–463.

6. Rumsfeld JS, Havranek E, Masoudi FA, Peterson ED, Jones P, Tooley JF,Krumholz HM, Spertus JA. Depressive symptoms are the strongest pre-dictors of short-term declines in health status in patients with heart failure.J Am Coll Cardiol. 2003;42:1811–1817.

7. Sullivan M, Simon G, Spertus J, Russo J. Depression-related costs inheart failure care. Arch Intern Med. 2002;162:1860–1866.

8. Rumsfeld JS, Jones PG, Whooley MA, Sullivan MD, Pitt B, WeintraubWS, Spertus JA. Depression predicts mortality and hospitalization inpatients with myocardial infarction complicated by heart failure. AmHeart J. 2005;150:961–967.

9. Junger J, Schellberg D, Muller-Tasch T, Raupp G, Zugck C, HaunstetterA, Zipfel S, Herzog W, Haass M. Depression increasingly predicts mor-tality in the course of congestive heart failure. Eur J Heart Fail. 2005;7:261–267.

10. Cleary PD, Edgman-Levitan S. Health care quality: incorporatingconsumer perspectives. JAMA. 1997;278:1608–1612.

11. Burke L. FDA Perspectives on IRT/CAT: DIA Workshop on Advances inHealth Outcomes Measurement: Exploring the Current State and theFuture Applications of Item Response Theory, Item Banks, andComputer-Adaptive Testing. Bethesda, MD: conference lecture; 2004.

12. McHorney CA, Cohen AS. Equating health status measures with itemresponse theory: illustrations with functional status items. Med Care.2000;38:II43–II59.

13. Rose M, Bezjak A. Logistics of collecting patient-reported outcomes(PROs) in clinical practice: an overview and practical examples. QualLife Res. 2009;18:125–136.

14. McHorney CA, Tarlov AR. Individual-patient monitoring in clinicalpractice: are available health status surveys adequate? Qual Life Res.1995;4:293–307.

15. Rector TS. A conceptual model of quality of life in relation to heartfailure. J Card Fail. 2005;11:173–176.

Table 3. Score Differences Between Different NYHA Classes

Variable No. of Items

NYHA Class*

Eta2 F Value P Value RV (95% CI)I (n�11) II (n�53) III/IV (n�36)

Physical disability 4.9�1.5 53.0 (6.2) 58.9 (8.6) 62.6 (7.4) 0.12 6.2 0.003 1.01 (0.38–2.20)

Fatigue 3.7�0.7 46.8 (6.9) 52.0 (7.6) 55.4 (9.4) 0.09 4.9 0.009 0.80 (0.21–1.91)

Dyspnea 4.6�1.5 43.9 (14.4) 53.8 (12.7) 59.8 (11.7) 0.13 6.9 0.002 1.13 (0.34–2.67)

MLHFQ 21 15.5 (14.8) 38.3 (25.3) 44.9 (22.9) 0.11 6.1 0.003 1.00

The theta values of the computer adaptive test (CAT) scales are scored on a T-distribution. The MLHFQ scores are summary scores ranging from 0 to 105. Allanalyses were controlled for the order of administration as a confounding variable. A bootstrap analysis was used to determine the CIs. NYHA indicates New YorkHeart Association; RV, relative validity (heart failure–CAT scale F values/F value for the MLHFQ sum scale); MLHFQ, Minnesota Living with Heart Failure Questionnaire.

*Data are given as mean (SD).

338 Circ Heart Fail May 2012

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

16. Garin O, Ferrer M, Pont A, Rue M, Kotzeva A, Wiklund I, Van GE,Alonso J. Disease-specific health-related quality of life questionnaires forheart failure: a systematic review with meta-analyses. Qual Life Res.2009;18:71–85.

17. Bjorner JB, Chang CH, Thissen D, Reeve BB. Developing tailoredinstruments: item banking and computerized adaptive assessment. QualLife Res. 2007;16(suppl 1):95–108.

18. Wainer H, Dorans NJ, Eignor D, Flaugher R, Green BF, Mislevy RJ,Steinberg L, Thissen D. Computerized Adaptive Testing: A Primer.Mahwah, NJ: Lawrence Erlbaum Associates; 2000.

19. Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, Ader D,Fries JF, Bruce B, Rose M. The Patient-Reported Outcomes MeasurementInformation System (PROMIS): progress of an NIH Roadmap coop-erative group during its first two years. Med Care. 2007;45:S3–S11.

20. Rose M, Bjorner JB, Becker J, Fries JF, Ware JE. Evaluation of apreliminary physical function item bank supported the expectedadvantages of the Patient-Reported Outcomes Measurement InformationSystem (PROMIS). J Clin Epidemiol. 2008;61:17–33.

21. Fliege H, Becker J, Walter OB, Bjorner JB, Klapp BF, Rose M. Devel-opment of a computer-adaptive test for depression (D-CAT). Qual LifeRes. 2005;14:2277–2291.

22. Rubin DB. Matched Sampling for Causal Effects. New York, NY: Cam-bridge University Press; 2006.

23. Ware JE Jr, Kosinski M, Bjorner JB, Bayliss MS, Batenhorst A, DahlofCG, Tepper S, Dowson A. Applications of computerized adaptive testing(CAT) to the assessment of headache impact. Qual Life Res. 2003;12:935–952.

24. Ware JE Jr, Dewey J. How to Score Version Two of the SF-36 HealthSurvey. Lincoln, RI: QualityMetric Inc; 2000.

25. National Committee for Quality Assurance. Specifications for theMedicare Health Outcomes Survey: HEDIS. 6. Washington, DC: NationalCommittee for Quality Assurance; 2004.

26. Rector T, Cohn J. Patients’ self-assessment of their congestive heartfailure, part 2: content, reliability and validity of a new measure, theMinnesota Living with Heart Failure questionnaire. Heart Failure. 1987;3:198–209.

27. Fliege H, Becker J, Walter OB, Rose M, Bjorner JB, Klapp BF. Eval-uation of a computer-adaptive test for the assessment of depression(D-CAT) in clinical application. Int J Methods Psychiatr Res. 2009;18:23–36.

28. Anatchkova MD, Saris-Baglama RN, Kosinski M, Bjorner JB. Devel-opment and preliminary testing of a computerized adaptive assessment ofchronic pain. J Pain. 2009;10:932–943.

29. Turner-Bowker DM, Saris-Baglama RN, Anatchkova M, Mosen DM. Acomputerized asthma outcomes measure is feasible for disease man-agement. Am J Pharm Benefits. 2010;2:119–124.

30. Schwartz C, Welch G, Santiago-Kelley P, Bode R, Sun X. Computerizedadaptive testing of diabetes impact: a feasibility study of Hispanics and

non-Hispanics in an active clinic population. Qual Life Res. 2006;15:1503–1518.

31. Chien TW, Wu HM, Wang WC, Castillo RV, Chou W. Reduction inpatient burdens with graphical computerized adaptive testing on the ADLscale: tool development and simulation. Health Qual Life Outcomes.2009;7:39.

32. Haley SM, Gandek B, Siebens H, Black-Schaffer RM, Sinclair SJ, TaoW, Coster WJ, Ni P, Jette AM. Computerized adaptive testing forfollow-up after discharge from inpatient rehabilitation: II: participationoutcomes. Arch Phys Med Rehabil. 2008;89:275–283.

33. Hart DL, Wang YC, Stratford PW, Mioduski JE. Computerized adaptivetest for patients with knee impairments produced valid and responsivemeasures of function. J Clin Epidemiol. 2008;61:1113–1124.

34. Hart DL, Werneke MW, Wang YC, Stratford PW, Mioduski JE. Com-puterized adaptive test for patients with lumbar spine impairmentsproduced valid and responsive measures of function. Spine (Phila Pa1976). 2010;35:2157–2164.

35. Gibbons RD, Weiss DJ, Kupfer DJ, Frank E, Fagiolini A, GrochocinskiVJ, Bhaumik DK, Stover A, Bock RD, Immekus JC. Using computerizedadaptive testing to reduce the burden of mental health assessment. Psy-chiatr Serv. 2008;59:361–368.

36. Walter OB, Becker J, Bjorner JB, Fliege H, Klapp BF, Rose M. Devel-opment and evaluation of a computer adaptive test for “Anxiety”(A-CAT). Qual Life Res. 2007;16(suppl 1):143–155.

37. Bayliss MS, Dewey JE, Dunlap I, Batenhorst AS, Cady R, Diamond ML,Sheftell F. A study of the feasibility of Internet administration of acomputerized health survey: the headache impact test (HIT). Qual LifeRes. 2003;12:953–961.

38. Bennett SJ, Oldridge NB, Eckert GJ, Embree JL, Browning S, Hou N,Chui M, Deer M, Murray MD. Comparison of quality of life measures inheart failure. Nurs Res. 2003;52:207–216.

39. Green CP, Porter CB, Bresnahan DR, Spertus JA. Development andevaluation of the Kansas City Cardiomyopathy Questionnaire: a newhealth status measure for heart failure. J Am Coll Cardiol. 2000;35:1245–1255.

40. Martin M, Kosinski M, Bjorner JB, Ware JE Jr, Maclean R, Li T. Itemresponse theory methods can improve the measurement of physicalfunction by combining the modified health assessment questionnaire andthe SF-36 physical function scale. Qual Life Res. 2007;16:647–660.

41. Haley SM, Fragala-Pinkham M, Ni P. Sensitivity of a computer adaptiveassessment for measuring functional mobility changes in childrenenrolled in a community fitness programme. Clin Rehabil. 2006;20:616–622.

42. Ruo B, Choi SW, Baker DW, Grady KL, Cella D. Development andvalidation of a computer adaptive test for measuring dyspnea in heartfailure. J Card Fail. 2010;16:659–668.

43. Becker J, Fliege H, Kocalevent RD, Bjorner JB, Rose M, Walter OB,Klapp BF. Functioning and validity of a Computerized Adaptive Test tomeasure anxiety (A-CAT). Depress Anxiety. 2008;25:E182–E194.

CLINICAL PERSPECTIVEPatient-reported outcome measures can assist clinicians in monitoring the effectiveness of their treatment of patients withheart failure (HF), comparing improvements under different forms of therapy, and identifying risk of deterioration. Thereare several questionnaires available to measure typical HF symptoms. However, established questionnaires are often longor too imprecise for individual decision making. New computer adaptive tests (CATs) promise to provide more precisemeasures, with fewer items, and an effective resolution to the classic conflict between practicality and precision faced bytraditional measurement methods. The CATs tailor each assessment to the individual’s status on what is being measured,applying only items that are most appropriate for her or his current health status. We have developed a CAT for theassessment of 3 typical HF symptoms (HF-CAT): dyspnea, fatigue, and physical function. In-clinic tests confirmed theexpected CAT advantages, including shorter surveys by eliminating questions not relevant to each patient, equal or betterenumerations over a wide range of scores, and surveys that did not require a test administrator. In summary, we havedeveloped a promising method to measure patient-reported symptoms for use in the care of patients with HF. This newmeasure is part of a rapidly growing number of new assessment tools using the advantages of item response theory andcomputerized adaptive test techniques.

Rose et al Heart Failure CAT 339

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from

Löwe, Thomas S. Rector and John E. WareMatthias Rose, Milena Anatchkova, Jason Fletcher, Arthur E. Blank, Jakob Bjørner, Bernd

Computerized Adaptive TestShort and Precise Patient Self-Assessment of Heart Failure Symptoms Using a

Print ISSN: 1941-3289. Online ISSN: 1941-3297 Copyright © 2012 American Heart Association, Inc. All rights reserved.

75231is published by the American Heart Association, 7272 Greenville Avenue, Dallas, TXCirculation: Heart Failure

doi: 10.1161/CIRCHEARTFAILURE.111.9649162012;5:331-339; originally published online April 23, 2012;Circ Heart Fail. 

http://circheartfailure.ahajournals.org/content/5/3/331World Wide Web at:

The online version of this article, along with updated information and services, is located on the

  http://circheartfailure.ahajournals.org//subscriptions/

is online at: Circulation: Heart Failure Information about subscribing to Subscriptions: 

http://www.lww.com/reprints Information about reprints can be found online at: Reprints:

  document. Permissions and Rights Question and Answer about this process is available in the

located, click Request Permissions in the middle column of the Web page under Services. Further information isthe Editorial Office. Once the online version of the published article for which permission is being requested

can be obtained via RightsLink, a service of the Copyright Clearance Center, notCirculation: Heart Failurein Requests for permissions to reproduce figures, tables, or portions of articles originally publishedPermissions:

by guest on May 22, 2018

http://circheartfailure.ahajournals.org/D

ownloaded from