Upload
hprctmu
View
260
Download
0
Embed Size (px)
Citation preview
Alan M. Jette, PT, PhD Health & Disability Research Institute
BU School of Public Health
Describe the measurement dilemma in Post-Acute Care (PAC)
Illustrate how Classical Test Theory methods have difficulty solving this dilemma
Introduce Contemporary Measurement Methods and illustrate how they can solve the measurement dilemma in PAC
The Centers for Medicare and Medicaid Services (CMS) has acknowledged that steps need to be taken toward developing a PAC system in the United States that provides payment and assures quality for an overall episode of PAC, rather than for each individual component.
As an essential step toward accomplishing this policy objective, assessment methods are needed to collect and compare relevant outcome and quality of care data across various sites where PAC is provided.
PAC treatment regimens and costs vary considerably between sites of service.
Little is known about which care setting provides the most
effective and cost/effective PAC due to difficulties in measuring functional outcomes across PAC settings using the same outcome metric.
►Administration:
A set of items (scale) is presented to participants
Responses to each item are assigned a number (item scores)
Item scores manipulated (usually totaled) to get a score for the measure
Focus is on “true score”
Proportion of correct responses or summated score –
Item-total correlations
Scale reliability (internal consistency)
Standard error of measurement
Measurement as a problem of search. CTT is linear in its approach
Suppose our subject is 73 on a 1 to a 100 scale…
In a CTT measure, each item in the measure is assessed, and a total score is calculated to determine where a person is located on the scale.
0 100 73
Classic Testing Theory
Found!
PAC outcome instruments are setting specific: FIM; MDS;OASIS;SF-36
Instruments differ in: # of items,
different response options, different time periods, or different item content; do not communicate with each other.
Need many items or many instruments to cover all the relevant functional outcomes across a broad range of PAC patients
The traditional administration of all
items to each person is burdensome & costly
1
3
1
3
5
7
Not Enough
Marks –
Too Noisy Too
Low - Ceiling
Effect
Example: FIM Example: PF-10
PF-10
Vigorous activities
Moderate activities
Moderate activities
Trouble bending, stooping
Need help to bathe
Cannot maintain balance
Move about with help
Stand up with help
ADL
FIM
PF-10
Source: Andres PL, Black-Schaffer R, Ni P, Haley SM. Computer adaptive testing: a strategy for monitoring stroke rehabilitation across settings. Topics in Stroke Rehabilitation ; 11(2); 33-39, 2004
Ideal Ruler
1
2
3
4
5
6
7
Self Care Eating Oral hygiene Toileting hygiene Shower/bathe self Upper body dressing Lower body dressing Putting on/taking off footwear Mobility Roll left & right Sit to lying Lying to sitting on side of the bed Sit to stand Chair/bed-to-chair transfer
Toilet transfer Car transfer Walk 10 feet Walk 50 feet Walk 150 feet Walk 10 ft on uneven surface 1 Step (curb) 4 steps 12 steps Picking up an object Wheel 50 ft. with 2 turns Wheel 150 ft.
Item difficulty with the hardest item (converse/Noisy)at the top and the easiest item (Converse/family)at the bottom.
Vertical spacing between the items on the hierarchy shows the approximate placement of the item on Communication (the trait) and corresponds to the
item difficulty estimate
The metric for this trait is along the bottom of the graphic and ranges from −7 to 7; the figure shows the relative placement of each item’s response levels on the ruler.
Each item is shown along the ruler where a “1” response for “unable” to “2” “a lot of difficulty” to “3” for “a little difficulty” and a “4” for “no difficulty”).
The expected responses for each item are denoted by their category value (eg., a value of “2” for a lot of difficulty), and the threshold between two levels of response is in the middle between the two responses usually represented by a colon (:). This threshold represents the 50 percent chance that a patient with a given ability has of responding in either a higher or lower category.
Figure 9. Mobility (N=2,980) Keyform (Admissions)
-4 -3 -2 -1 0 1 2 3 4
|------+------+------+------+------+------+------+------| NUM ITEM
1 1 : 2 : 3 : 4 : 5 : 6 6 66 III_F10_funcmob12steps
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 67 III_F11_funcmobupdownramp
1 1 : 2 : 3 : 4 : 5 : 6 6 62 III_F6_funcmobpickup
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 64 III_F8_funcmobwalkuneven
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 65 III_F9_funcmob4steps
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 68 III_F12a_funcmobwalk500
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 63 III_F7_funcmobwalk50turn
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 59 III_F3_funcmoblie2sit
1 1 : 2 : 3 : 4 : 5 : 6 6 72 III_F13a_funcmobwheel500
1 1 : 2 : 3 : 4 : 5 : 6 6 60 III_F4_funcmobsit2stand
1 1 : 2 : 3 : 4 : 5 : 6 6 75 III_F13d_funcmobwheelroom
1 1 : 2 : 3 : 4 : 5 : 6 6 71 III_F12d_funcmobwalkroom
1 1 : 2 : 3 : 4 : 5 : 6 6 70 III_F12c_funcmobwalk50
1 1 : 2 : 3 : 4 : 5 : 6 6 69 III_F12b_funcmobwalk150
1 1 : 2 : 3 : 4 : 5 : 6 6 61 III_F5_funcmobchairtrans
1 1 : 2 : 3 : 4 : 5 : 6 6 57 III_F1_funcmobsitlie
1 1 : 2 : 3 : 4 : 5 : 6 6 58 III_F2_funcmobrolllr
1 1 : 2 : 3 : 4 : 5 : 6 6 74 III_F13c_funcmobwheel50
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 73 III_F13b_funcmobwheel150
|------+------+------+------+------+------+------+------| NUM ITEM
-4 -3 -2 -1 0 1 2 3 4
1
1 2 0
21 233244545567776445635935626168
1 3 1 1 321346879798554857173224955669233176958334 PERSON
T S M
0 10 20 30 40 50 60 99 PERCENTILE
A patient at ‘1’ on the ruler aligns perfectly with the “6” category response for the easiest item, (wheelchair mobility) , and therefore that person has a very high likelihood of responding (no difficulty) in wheelchair mobility
Figure 9. Mobility (N=2,980) Keyform (Admissions)
-4 -3 -2 -1 0 1 2 3 4
|------+------+------+------+------+------+------+------| NUM ITEM
1 1 : 2 : 3 : 4 : 5 : 6 6 66 III_F10_funcmob12steps
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 67 III_F11_funcmobupdownramp
1 1 : 2 : 3 : 4 : 5 : 6 6 62 III_F6_funcmobpickup
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 64 III_F8_funcmobwalkuneven
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 65 III_F9_funcmob4steps
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 68 III_F12a_funcmobwalk500
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 63 III_F7_funcmobwalk50turn
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 59 III_F3_funcmoblie2sit
1 1 : 2 : 3 : 4 : 5 : 6 6 72 III_F13a_funcmobwheel500
1 1 : 2 : 3 : 4 : 5 : 6 6 60 III_F4_funcmobsit2stand
1 1 : 2 : 3 : 4 : 5 : 6 6 75 III_F13d_funcmobwheelroom
1 1 : 2 : 3 : 4 : 5 : 6 6 71 III_F12d_funcmobwalkroom
1 1 : 2 : 3 : 4 : 5 : 6 6 70 III_F12c_funcmobwalk50
1 1 : 2 : 3 : 4 : 5 : 6 6 69 III_F12b_funcmobwalk150
1 1 : 2 : 3 : 4 : 5 : 6 6 61 III_F5_funcmobchairtrans
1 1 : 2 : 3 : 4 : 5 : 6 6 57 III_F1_funcmobsitlie
1 1 : 2 : 3 : 4 : 5 : 6 6 58 III_F2_funcmobrolllr
1 1 : 2 : 3 : 4 : 5 : 6 6 74 III_F13c_funcmobwheel50
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 73 III_F13b_funcmobwheel150
|------+------+------+------+------+------+------+------| NUM ITEM
-4 -3 -2 -1 0 1 2 3 4
1
1 2 0
21 233244545567776445635935626168
1 3 1 1 321346879798554857173224955669233176958334 PERSON
T S M
0 10 20 30 40 50 60 99 PERCENTILE
The person distribution of ‘Mobility’ constitutes the three rows above and next to the PERSON label. Each column creating the person distribution represents the number of persons at a given point on the ruler.
Figure 9. Mobility (N=2,980) Keyform (Admissions)
-4 -3 -2 -1 0 1 2 3 4
|------+------+------+------+------+------+------+------| NUM ITEM
1 1 : 2 : 3 : 4 : 5 : 6 6 66 III_F10_funcmob12steps
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 67 III_F11_funcmobupdownramp
1 1 : 2 : 3 : 4 : 5 : 6 6 62 III_F6_funcmobpickup
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 64 III_F8_funcmobwalkuneven
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 65 III_F9_funcmob4steps
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 68 III_F12a_funcmobwalk500
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 63 III_F7_funcmobwalk50turn
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 59 III_F3_funcmoblie2sit
1 1 : 2 : 3 : 4 : 5 : 6 6 72 III_F13a_funcmobwheel500
1 1 : 2 : 3 : 4 : 5 : 6 6 60 III_F4_funcmobsit2stand
1 1 : 2 : 3 : 4 : 5 : 6 6 75 III_F13d_funcmobwheelroom
1 1 : 2 : 3 : 4 : 5 : 6 6 71 III_F12d_funcmobwalkroom
1 1 : 2 : 3 : 4 : 5 : 6 6 70 III_F12c_funcmobwalk50
1 1 : 2 : 3 : 4 : 5 : 6 6 69 III_F12b_funcmobwalk150
1 1 : 2 : 3 : 4 : 5 : 6 6 61 III_F5_funcmobchairtrans
1 1 : 2 : 3 : 4 : 5 : 6 6 57 III_F1_funcmobsitlie
1 1 : 2 : 3 : 4 : 5 : 6 6 58 III_F2_funcmobrolllr
1 1 : 2 : 3 : 4 : 5 : 6 6 74 III_F13c_funcmobwheel50
| |
| |
1 1 : 2 : 3 : 4 : 5 : 6 6 73 III_F13b_funcmobwheel150
|------+------+------+------+------+------+------+------| NUM ITEM
-4 -3 -2 -1 0 1 2 3 4
1
1 2 0
21 233244545567776445635935626168
1 3 1 1 321346879798554857173224955669233176958334 PERSON
T S M
0 10 20 30 40 50 60 99 PERCENTILE
The letters underneath the person distribution indicate the placement of the average person ability estimate (M), and one and two times the standard deviation around that estimate (S and T respectively).
Item Response Theory (IRT) Methodology & Computer Adaptive Testing….
Ideal Ruler
1
2
3
4
5
6
7
Patient
scores
here
Match Questions to the Respondent
=
a probability theory of measurement stating that the correct answer to an
item (question) is a function of an underlying ability of
the person and the characteristics of that item.
Large pools of functional status items are selected/written to cover the entire continuum of an outcome domain. Items in the pool provide different information about the outcome across the continuum. Items from other instruments can be included Scores are calibrated to an equal interval score. Reliability & precision are estimated across the continuum
Physical Functioning Item Bank
Item
1
Item
2
Item
3
Item
4
Item
5
Item
6
Item
7
Item
8
Item
9
Item
n
100500
Are you able to get in and out of bed?
Are you able to stand without losing your balance for 1 minute?
Are you able to walk from one room to another?
Are you able to walk a block on flat ground?
Are you able to run or jog for two miles?
Are you able to run five miles?
Physical Function
Physical Functioning Item Bank
Item
1
Item
2
Item
3
Item
4
Item
5
Item
6
Item
7
Item
8
Item
9
Item
n
100500
Are you able to get in and out of bed?
Are you able to stand without losing your balance for 1 minute?
Are you able to walk from one room to another?
Are you able to walk a block on flat ground?
Are you able to run or jog for two miles?
Are you able to run five miles?
Physical Functioning Item Bank
Item
1
Item
2
Item
3
Item
4
Item
5
Item
6
Item
7
Item
8
Item
9
Item
n
100500
Are you able to get in and out of bed?
Are you able to stand without losing your balance for 1 minute?
Are you able to walk from one room to another?
Are you able to walk a block on flat ground?
Are you able to run or jog for two miles?
Are you able to run five miles?
Physical Functioning Item Bank
Item
1
Item
2
Item
3
Item
4
Item
5
Item
6
Item
7
Item
8
Item
9
Item
n
100500
Are you able to get in and out of bed?
Are you able to stand without losing your balance for 1 minute?
Are you able to walk from one room to another?
Are you able to walk a block on flat ground?
Are you able to run or jog for two miles?
Are you able to run five miles?
Physical Functioning Item Bank
Item
1
Item
2
Item
3
Item
4
Item
5
Item
6
Item
7
Item
8
Item
9
Item
n
100500
Are you able to get in and out of bed?
Are you able to stand without losing your balance for 1 minute?
Are you able to walk from one room to another?
Are you able to walk a block on flat ground?
Are you able to run or jog for two miles?
Are you able to run five miles?
Physical Functioning Item Bank
Item
1
Item
2
Item
3
Item
4
Item
5
Item
6
Item
7
Item
8
Item
9
Item
n
100500
Are you able to get in and out of bed?
Are you able to stand without losing your balance for 1 minute?
Are you able to walk from one room to another?
Are you able to walk a block on flat ground?
Are you able to run or jog for two miles?
Are you able to run five miles?
A tailored, individualized measure which involves
selecting a set of test items from a calibrated item pool for each patient that best
measures the ‘ability’ of that person
Integrates IRT with computers to administer a measure
selects questions on the basis
of a patient’s response to previously administered questions
measurement is “adapted” to individual
skips uninformative items to minimize response burden
allows determination of person’s standing on a domain without a loss in measurement precision.
Measurement as a problem of search. IRT/CAT measurement is binary in its approach
Suppose our subject is 73 on a 1 to 100 scale…
IRT with CAT
0 100 73
Found!
32
Includes three patient-reported outcome scales, or domains.
Basic Mobility Daily Activities Applied Cognitive
34
•3 quantitative PRO scales; each takes 2-3 minutes to complete & score •Assesses assistance or difficulty in functional performance •Developed with Federal Government funding •Developed & validated in a sample of 1041; selected from IRF, SNF, LTCH, HHA, outpatient services •Available in over 20 languages Jette, Haley, et al: Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. PHYSICAL THERAPY 87(4): 385-398,2007 Haley,Jette, et al. Replenishing a Computerized Adaptive Test (CAT) of Patient Reported Outcomes. Quality of Life Research, 18(4) 461-471, 2009.
35
Activity Measure for Post-acute
Care (AM-PAC) 269 items
Basic Mobility 131 items
Applied Cognition 50 items
Daily Activities 88 items
Ambulation
Bend, Carry, Lift
Locomotion with Device
Meal Preparation
Grooming/Dressing
Feeding Communication
Print Information
Complex Instructions
Transfer
Provides a quantitative score
Can compare functioning within an individual (admission to discharge) and across individuals
Can compare across an episode of care: transitions from inpatient – to outpatient – to home settings
Carefully selected items from the AM-PAC calibrated item banks. Need to assess different kinds of activities in
inpatient and outpatient settings ▪ Developed separate short forms for each setting
Inpatient and outpatient short form items include items calibrated on the same metric ▪ Scores from inpatient and outpatient short forms assess
patients’ function as they move from inpatient to outpatient settings.
-3
-2
-1
0
1
2
3Daily Activity Short Form Item Difficulty N = 88
Selected 6 Items for Inpatient; 15 Items for Outpatient
Inpatient
Outpatient
Table 5a. CARE-C and CARE-F Analytic Subscale Information: Patient Self-Report
Subscale Item Content Variable Name
CARE-C
Basic Mobility How much difficulty do you currently have…
Moving from sitting at the side of the bed to
lying down on your back?
II_E1a_patmobbed2back
Moving up in bed? II_E1b_patmobupbed
Standing for at least one minute? II_E1c_patmobstand1min
Sitting down in an armless straight chair? II_E1d_patmobsitchair
Standing up from an armless straight chair? II_E1e_patmobstandchair
Getting into and out of a car/taxi? II_E1f_patmobcar
Walking around on one floor, taking into
consideration thresholds, doors, furniture,
and a variety of floor coverings?
II_E1g_patmobwalkonefloor
Going up and down a flight of stairs inside,
using a handrail?
II_E1h_patmobstairs
How much difficulty do you currently have…
Bending over from a standing position to
pick up a piece of clothing from the floor
without holding onto anything?
II_E1i_patmobbend
Walking several blocks? II_E1j_patmobwalkblocks
Walking up and down steep unpaved
inclines?
II_E1k_patmobincline
Carrying something in both arms while
climbing a flight of stairs?
II_E1l_patmobcarrystairs
How much help from another person do you
currently need moving to and from a bed to
a chair?
II_E1m_patmobbed2chair
Figure 1. Basic Mobility (N=2,277) Keyform (Admissions)
-6 -4 -2 0 2 4 6 8
|-------+-------+-------+-------+-------+-------+-------| NUM ITEM
0 0 : 1 : 2 : 3 3 12 II_E1l_patmobcarrystairs
| |
| |
0 0 : 1 : 2 : 3 3 11 II_E1k_patmobincline
0 0 : 1 : 2 : 3 3 10 II_E1j_patmobwalkblocks
| |
| |
0 0 : 1 : 2 : 3 3 8 II_E1h_patmobstairs
0 0 : 1 : 2 : 3 3 9 II_E1i_patmobbend
| |
0 0 : 1 : 2 : 3 3 5 II_E1e_patmobstandchair
0 0 : 1 : 2 : 3 3 6 II_E1f_patmobcar
| |
0 0 : 1 : 2 : 3 3 2 II_E1b_patmobupbed
0 0 : 1 : 2 : 3 3 7 II_E1g_patmobwalkonefloor
0 0 : 1 : 2 : 3 3 1 II_E1a_patmobbed2back
0 0 : 1 : 2 : 3 3 4 II_E1d_patmobsitchair
0 0 : 1 : 2 : 3 3 3 II_E1c_patmobstand1min
| |
| |
00 : 1 : 2 : 3 3 13 II_E1m_patmobbed2chair
|-------+-------+-------+-------+-------+-------+-------| NUM ITEM
-6 -4 -2 0 2 4 6 8
1 1 11111111
1 1 3334057480821202537767 51213 11 4
3 21 41416608670497497789687003304465720860234711 25 PERSON
T S M S T
0 10 20 40 50 70 80 90 99 PERCENTILE
A typical CAT takes 2-3 minutes to administer Avoids unnecessary items Adequate reliability and sensitivity Ability to cover wide content scope Score reports can be developed to assist in
interpretation
SCI-FI/AT raw scores are converted to standardized z-scores normed on the full item bank which are then transformed to ‘T’-scores such that the mean is 50 and the standard deviation is 10.
AM-PAC Short Forms and CATs are all on the same common ‘T’ scale to facilitate functional assessment across PAC settings
Requires IT platforms unavailable in some rehabilitation settings or use of short forms
Legacy instruments are required by regulation Discomfort in asking different items over time and across
patients Create vehicles for clinician training & support Create an system allowing access to item banks and CATs
IRT and CAT technologies could
transform outcome assessment in PAC Contemporary outcomes measures like
AM-PAC have great potential for overcoming the measurement dilemma in PAC
Challenges to adopting measurement
innovations like IRT/CAT need to be overcome.
Jette A, Haley S: Contemporary measurement techniques for rehabilitation outcome assessment. Journal of Rehabilitation Medicine. 37(6):339-345, 2005.
Haley, S.M., Coster, W.J., Andres, P.L., Ludlow, L.H., Bond, T., Sinclair, S.J., & Jette, A.M. (2004). Activity Outcome Measurement for Post-acute Care. Medical Care, 42(1 Suppl), I-49-I-69.
Jette, A.M., Haley, S., Tao, W., Ni, P., Meyers, D., & Zurek, M. (2007). Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Physical Therapy, 87(4), 385-98.
Jette, A.M., Norweg, A., & Haley, S. (2008). Achieving meaningful measurements of ICF concepts. Disability and Rehabilitation, 30(12-13), 963-9.
Jette, A.M., Haley, S.M., Ni, P., & Moed, R. (2008). Adaptive short forms for outpatient rehabilitation outcome assessment. American Journal of Physical Medicine and Rehabilitation, 87(10), 842-52.
Jette, D.U., Stilphen, M., Ranganathan, V.K., Passek, S.D., Frost, F.S., & Jette, A.M. (2014). AM-PAC “6-Clicks” functional assessment scores predict acute care hospital discharge destination. Physical Therapy, 94(9), 1252-61.
•Jette, A.M., Haley, S., Tao, W., Ni, P., Meyers, D., & Zurek, M. (2007). Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Physical Therapy, 87(4), 385-98.
•Jette, A.M., Haley, S., Tao, W., Ni, P., Meyers, D., & Zurek, M. (2007). Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Physical Therapy, 87(4), 385-98.