M i th P ti t E i t thMeasuring the Patient Experience at the Physician Group and Doctor Levels:

On the Ground in California

Cheryl Damberg, Ph.D.Senior Researcher, RAND

January 27 2011January 27, 2011CHCF Conference: Transforming Healthcare Through the Patient Experience“The Science of Measurement—Collecting Meaningful and Actionable Data”

Background on Patient Experience Surveying in California

• 1996 and 1998 Pacific Business Group on Health• 1996 and 1998 Pacific Business Group on Health (PBGH) launched a patient experience survey in 58 medical groups in Californiag p

– Also measured 2-year changes in patient functional status (SF-12)

• Found better quality of care resulted in slower functional decline*

• Redundant and conflicting survey efforts by the• Redundant and conflicting survey efforts by the health plans led to consolidated effort to use common patient experience survey to assesscommon patient experience survey to assess medical group performance

*KL Kahn DM Tisnado JL Adams H Liu W Chen F Hu CM Mangione RD Hays and Cheryl

2 1/27/2011

KL Kahn, DM Tisnado, JL Adams, H Liu, W Chen, F Hu, CM Mangione, RD Hays, and Cheryl L. Damberg. Does Ambulatory Process of Care Predict Health-Related Quality of Life Outcomes for Patients with Chronic Disease? Health Services Research 42:1, Part I (February 2007)

Patient Experience Measurement C t L d i C lif iCurrent Landscape in California

P ti tTransparency:

Patient Assessment

Survey Group

p yOffice of the Public

Advocate Consumer Website(PAS)

Group-level S

Group results

Website

Pay-for-Performance:Survey yIHA P4P Program

Q lit I tDoctor-level

Surveys

Quality Improvement:Physician Groups

California Quality Collaborative Doctor-

Level PAS

y(CQC)

Pay-for-Performance;

3 1/27/2011

Pay-for-Performance;Medical Group Internal Incentives

Surveying the Patient Experience at the GPhysician Group Level

• Since 2001, the California Collaborative Healthcare Reporting Initiative (CCHRI) has been annually fielding a patient experience survey to assess the care in ~140 medical groups statewidemedical groups statewide

– Results are publicly reported• Use the Patient Assessment Survey (PAS)y ( )

– A derivative of the Clinician-Group CAHPS survey• 2010 Survey:

– Commercially-insured HMO/POS enrollees, ages >=18• Randomly sampled 900 adult patients per group

– n=450 PCPs and n=450 specialists– n=450 PCPs and n=450 specialists– 139 medical groups (179 reporting units)– 37.2% overall response rate

4 1/27/20114

Key Findings for 2010 PAS• Performance gains

More groups highly rated by the Office of the– More groups highly rated by the Office of the Public Advocate (OPA) public report card

– Steady small score increases 5 yrs– Steady, small score increases 5 yrs– Lowest quartile groups improve

• Performance variation• Performance variation– 5-10 point score range across groups

Lowest scoring groups close gap– Lowest scoring groups close gap– Variation at practice site and MD levels

• Performance gaps & opportunities• Performance gaps & opportunities– Access and chronic care self-mgm’t

Higher performance outside CA5 1/27/20115

– Higher performance outside CA

Medical Group Performance AdvancespPublic Reporting: Office of Patient Advocate

R ti 2010 2006Ratings— 2010 vs. 2006

Performance Ratings Performance Ratings% Medical Groups

2010% Medical Groups

2006Excellent 15% 1%15% 1%Good 66% 59%Fair 19% 39%Poor 0 1%

6 1/27/2011

Steady Small Gains in Statewide Average PerformanceAverage Performance

7 1/27/2011

+1.6 pts +2.5 pts +4.1 pts+2.1 pts

Lowest scoring groups see larger score gain compared to highest scoring groupsgain compared to highest scoring groups

over 5 years (2006 -2010)

8 1/27/2011

Performance Variation Across Medical Groups20102010

Mean 10th PCT 90th PCT

Patient-Doctor Interactions 89 86 91Office Staff 86 83 89Coordinated Care 76 71 81Patient Access 76 70 80Chronic Care 69 64 73Health Promotion 57 53 60

9 1/27/2011

Why focus on Doctor Level?Variation at Group Practice Site and PhysicianVariation at Group, Practice Site and Physician

LevelsOne medical group example: patient access

Medical Group “A”Access Mean Score

Practice SitesAccess Mean

PhysiciansAccess Mean Score

g p p p

ScoreMedical Group 70 PED Site 1 80

Spec Site 2 73Top 5 MDsMD1 93Spec Site 2 73

PCP Site 3 72PCP Site 4 71PCP Site 5 67

MD1 93MD2 88MD3 88MD4 88PCP Site 5 67

PCP Site 6 67PED Site 7 65OBG Site 8 64

MD4 88MD 5 88

Low 5 MDsOBG Site 8 64 Low 5 MDsMD6 53MD7 52MD8 50

10 1/27/2011

MD9 48MD10 42

Desire to Close the Gap Between CA and Other CG CAHPS Sites (Mean Scores)Other CG-CAHPS Sites (Mean Scores)

US National

‘05/06

NY ‘06 Med Center

Mass ‘07

CA ‘10

05/06

Needed care soon enough 88.1 83.9 73.9 85.0 79.8Regular appt. soon enough 92.2 87.5 84.6 83.1 83.4Wait less than 15 minutes 74.1 64.2 66.9 71.3 64.6Regular hrs call-back 86.8 79.9 71.8 82.3 77.8After hrs care met needs 84.1 80.4 70.2 82.0 73.4

Doctor explains things 94 6 92 4 95 4 92 9 90 2Doctor explains things 94.6 92.4 95.4 92.9 90.2Doctor listens carefully 95.2 93.5 96.1 92.3 90.0Instructions for symptoms 94.9 91.1 94.4 92.1 89.6Doctor knows medical hist 87.7 91.7 87.8 86.0Doctor spends enough time 92.2 90.5 92.2 88.7 86.7

11 1/27/2011

Doctor shows respect 96.3 94.6 96.8 93.2 91.1

Adjustment is limited to what was common among these datasets, age, sex, race, general health, but not chronic conditions.

2010 Doctor-Level Survey2010 Doctor-Level Survey• Sponsored by CCHRI, as an extension of the group-

level Patient Assessment Survey (PAS)– Designed to help groups reduce variation and improve scores on

statewide P4P measuresstatewide P4P measures• 29 physician groups participated

– Survey was mailed to 158,000 patientsy p• Response rate: 35% (over 56,000 patients)• ~35 or more patients of each physician completed the survey

• Survey modes: 2 mailings (or could complete on-line through web link)R lt fid ti l ( d i t ll b th• Results were confidential (used internally by the medical groups and their physicians)

12 1/27/2011

Sample Report Card to Doctors

13 1/27/2011

Sample Physician Report CardSample Physician Report Card

14 1/27/2011

The 21st Century: Sites Displaying Doctor Ratings Continue to Grow and EvolveContinue to Grow and Evolve

15 1/27/2011

M h d I C idMethods Issues to Consider

Psychometric properties of MeasuresReliability

ValidityRisk of Misclassification

16 1/27/2011

Psychometric Properties: 2010 Doctor SurveyRange of

Item-Scale Correlations

% Ceiling

Estimated MD-level reliability

( 35)

Minimum N required to

achieve 0.70 MD-l l li bilit

25th – 75th

range of means

MD(n=35) level reliability across MDs

Adult PCPNPT= 44,519N =1 111NMD=1,111

Quality of MD-Patient Interactions(n=6 items)

0.77-0.88 46.3 0.77 25 85.3-93.1

(n 6 items)

Health Promotion(n=2 items)

0.71-0.71 40.0 0.72 33 59.8-73.8

Access 0 50 0 70 14 8 0 87 12 69 3 82 8Access(n=5 items)

0.50-0.70 14.8 0.87 12 69.3-82.8

Coordination of Care(n=2 items)

0.50-0.50 46.1 0.77 25 72.0-85.3(n=2 items)

Chronic Care (n=5 items)

0.46-0.63 22.9 0.62 51 64.6-76.4

Offi St ff 0 81 0 81 53 2 0 78 23 80 8 90 8

17 1/27/2011

Office Staff(n=2 items)

0.81-0.81 53.2 0.78 23 80.8-90.8

Methods Issue: Reliability*

• A statistical concept that describes how well one can confidently distinguish the performance of oneconfidently distinguish the performance of one provider from another

• Measured as the ratio of the “signal” to the “noise”– The between-provider variation in performance is the

“signal”– The within-provider measurement error is the “noise”– Measured on a 0.0 to 1.0 scale

• Zero = all variability is due to noise or measurement error• 1 0 = all the variability is due to real differences in• 1.0 = all the variability is due to real differences in

performance

18 1/27/2011

*Source: Adams JL, The Reliability of Provider Profiling: A Tutorial, Santa Monica, Calif.: RAND Corporation,TR-653-NCQA, 2009. As of June 8, 2010: http://www.rand.org/pubs/technical_reports/TR653/

Between-Provider Performance VariationBetween-Provider Performance Variation

Lower between-provider variation(harder to tell who is best)

0 10050

Higher between-provider variation( i t t ll h i b t)(easier to tell who is best)

0 100500 10050

19 1/27/2011

= average performance for each provider

Different Levels of Measurement ErrorDifferent Levels of Measurement Error(Uncertainty about the “true” average performance)

Higher measurement error (harder to tell who is best)

( i ll h i b )

0 10050

Lower measurement error (easier to tell who is best)

0 10050

= average performance for

50

= range of uncertainty about “true” average

each provider performance

20 1/27/2011

Link between Reliability* and Risk ofLink between Reliability and Risk of Misclassification

Hi h li bilit i• Higher reliability in a measure:– Means more signal, less noise– Reduces likelihood that you will classify

provider in “wrong” category• Reliability is function of:

– Provider-to-provider variation– Sample size

• Per Adams*: “Reliability ASSUMES validity”

*Adams JL, The Reliability of Provider Profiling: A Tutorial, Santa Monica, Calif.: RAND Corporation,TR-653-NCQA, 2009. As of June 8, 2010:

21 1/27/2011

p , Q , ,http://www.rand.org/pubs/technical_reports/TR653/

Validityy

• Validity –the extent to which the performance information means what it is supposed to mean, rather than meaning something else– Ask yourself: Does the measurement measure

what it claims to measure?• Consider whether these threats to validity exist:

– Is the measure controllable by the provider?– Does patient behavior affects the measure?– Is the measure affected by differences in the patients

b i t t d ( i )?being treated (case mix)?– Is the measure controlled by other factors than the

provider?

22 1/27/2011

provider?

Good Resource for Provider MeasurementGood Resource for Provider Measurementand Reporting

• AHRQ Report:– Friedberg MW, Damberg CL. Methodological

Considerations in Generating Provider Performance Scores for Use in Public ReportingPerformance Scores for Use in Public Reporting by Chartered Value Exchanges. Rockville, MD: Agency for Healthcare Research and Quality; 2010.Agency for Healthcare Research and Quality; 2010.

• To obtain a copy of the White Paper, please email peggy.mcnamara@ahrq.hhs.gov and she will put you on the distribution list

23 1/27/2011

Measuring the Patient Experience:Collecting Standardized DataCollecting Standardized Data

Julie BrownDirector, Survey Research GroupDirector, Survey Research Group

January 27, 2011

What is Standardized Data Collection?What is Standardized Data Collection?

• Scientific sample of patients• Scientific sample of patients– No systematic bias in inclusion or exclusion of

patientspatients– Census or random sample

• The same data collection procedures are employed each time a clinic collects patient experience data

– Allows for trending of data– Promotes comparison of data across clinics*p

* When all the clinics use the same data collection procedures, same experience of care measures

2

Challenges in Collecting DataChallenges in Collecting Data

• Availability of survey tools• Availability of survey tools– Homegrown survey vs. standardized measures

• Availability of data collection tools– “How to” help in implementing an experience ofHow to help in implementing an experience of

care survey

• Cost– Traditional approach of a vendor-conducted

survey is perceived as too costly

3

Common Approaches to Data CollectionCommon Approaches to Data Collection

• In-office distribution of paper survey• In-office distribution of paper survey– Pros: Use existing staff (convenient), minimal burden on

daily work flow, get results quickly, minimal costC C fid ti lit i h ti t t ff– Cons: Confidentiality issues when patient care staff deliver survey, errors in survey delivery (who/when), burden on work flow (competing demands), cost is not minimalminimal

• Vendored mail surveyff– Pros: No burden to clinic staff, standardization,

benchmarking reports– Cons: Clinic population may have unstable housing,

takes too long to get results, cost

4

Improving In-Office Distribution of SurveyImproving In-Office Distribution of Survey

• Staff with no role in care delivery dedicated to• Staff with no role in care delivery dedicated to survey distribution

• Focus on delivering survey invitation to every eligible patient

• Visit-focused survey delivered after visit (staff stationed at exit point)stationed at exit point)

• Locked ballot box or mail return of surveyLocked ballot box or mail return of survey

We conducted an experiment in 6 California clinics

5

Lessons Learned: In-Office Survey DistributionLessons Learned: In Office Survey Distribution

• Even with dedicated staff it is a challenge to implement a• Even with dedicated staff, it is a challenge to implement a scientific approach to sampling when surveys are distributed at the clinic

Multiple exit points– Multiple exit points– Errors in timing of survey delivery– Trending and comparisons across clinics are at risk

• Cost of distributing surveys in the clinic is not minimal and not less than a vendored survey

• Percentage of patients who return a completed survey ranges from 21%-48% of those who accept a survey (how does that

il d ?)compare to mailed survey?)

6

Costs of Data Collection (Per Clinic)Costs of Data Collection (Per Clinic)

M d f D t Th W k f O W k fMode of Data Collection

Three Weeks of Patient Visits

One Week of Patient Visits

In clinic distribution of

$9,050 $4,019

paper survey

In clinic distribution of web

$8,760 $3,729distribution of web survey URL (paper back-up)Vendor conducts mailed survey

$5,777 $5,777

7

The Landscape of Patient The Landscape of Patient Experience MeasurementExperience Measurement

Dale Shaller, MPASh ll C l i GShaller Consulting GroupJanuary 27, 2011

Forces Driving MeasurementgPublic reporting

AF4Q and CVE initiativesState mandatesCMS Hospital and PhysicianCompare

Pay-for-performancePay-for-performancePatient-Centered Medical HomeHRSA Bureau of Primary Health CareAmerican Board of Medical SpecialtiesAmerican Board of Medical SpecialtiesRising consumer and patient expectations

2

Strategies for MeasurementgPatient surveysy

Home-grown surveysProprietary tools (most focus on “satisfaction”)Proprietary tools (most focus on satisfaction )Public domain instruments (CAHPS)

Focus groups and interviewsFocus groups and interviewsWalkthroughsShadowing“Mystery shopping”Mystery shoppingUser-posted online ratings and reviews

3

CAHPS Family of Surveysy yAmbulatory Care Surveysy y

CAHPS Clinician & Group SurveyCAHPS Health Plan SurveyCAHPS Health Plan SurveyCAHPS Surgical Care SurveyCAHPS Home Health Care SurveyCAHPS Home Health Care Survey

Facility SurveysCAHPS Hospital Survey (HCAHPS)CAHPS In-Center Hemodialysis SurveyCAHPS Nursing Home Survey

4

Core CAHPS Design Principlesg pFocus on topics for which consumers are the b l f fbest or only source of information

Include patient reports and ratings of Include patient reports and ratings of experiences – not “satisfaction”

Base question items and survey protocols on rigorous scientific development and testing, g p gas well as extensive stakeholder input

All d i i th bli All surveys and services are in the public domain

5

CAHPS Clinician & Group SurveyMultiple versions to meet user needs

p yp

Visit version12 th i12-month versionPatient-centered medical home (PCMH) versionAdult and child versions

Core questions are the same across versionsCore questions are the same across versionsSupplemental questions can be added for specific topics

6

CG-CAHPS PCMH Surveyy

CAHPS Clinician &

CAHPS and Other CAHPS

Group Core Questionnaire*

Supplemental Items

PCMH Survey

* NQF endorsed

7

PCMH Survey DomainsyAccess to careCommunication

About care from other providers (e.g., specialists)Among others at the provider’s office (e.g., care team)

CoordinationComprehensivenessShared decision-makingShared decision makingWhole person orientationS lf t tSelf-management support

8

CAHPS User Network ResourcesSurvey and Reporting Kitsy p gCAHPS DatabaseTechnical assistanceTechnical assistance

cahps1@ahrq.gov1 800 492 92611-800-492-9261

Conferences and webcastsCAHPS Improvement Guide

www.cahps.ahrq.govp q g

9

1 of 6

Overview of the CAHPS® Clinician & Group Survey The Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys are free,

nonproprietary instruments designed to support standardized measurement of the experiences of patients and health plan enrollees with care in a variety of settings. They have been developed by prominent research organizations under the auspices of the U.S. Agency for Healthcare Research and Quality. Learn more at www.cahps.ahrq.gov. One of the newest additions to the CAHPS suite of instruments is the CAHPS Clinician & Group Survey, which was endorsed by the National Quality Forum in July 2007.

Instruments Included in the Clinician & Group Survey The survey includes several instruments designed to enable users to assess and report on the experiences of adults and children in primary and specialty care settings:

• Adult Primary Care Questionnaire 1.0 • Adult Specialty Care Questionnaire 1.0 • Adult Visit Questionnaire for Primary and Specialty Care (beta) • Child Primary Care Questionnaire 1.0 • Child Primary Care Questionnaire 2.0 (beta)

Core and Supplemental Items in the Clinician & Group Survey Every CAHPS survey is composed of core items and supplemental items.

• The purpose of the required core items is to have a single set of questions that is consistent across all versions of the survey and lends itself to comparisons of validated composite measures across entities. For example, the Adult Primary Care Questionnaire 1.0 and Adult Specialty Care Questionnaire 1.0 consist of the same core items.

• The supplemental items offer users a variety of questions that can be added to the core items in order to address specific topics of interest. A combination of core and supplemental items will provide a customized CAHPS Clinician & Group Survey to meet your needs.

Overview of the CAHPS® Clinician & Group Survey

2 of 6

The table below shows the reporting measures and their items from the core instrument. Comparative data for these measures will be available through the CAHPS Database.

CAHPS Clinician & Group Survey Core Reporting Measures: Composites and Rating Getting appointments and health care when needed Got appointment for urgent care as soon as needed Got appointment for check-up or routine care as soon as needed Got answer to medical question the same day you phoned doctor’s office Got answer to medical question as soon as you needed when phoned doctor’s office after hours Saw doctor within 15 minutes of appointment time How well doctors communicate Doctor explained things in a way that was easy to understand Doctor listened carefully Doctor gave easy to understand instructions about taking care of health problems or concerns Doctor knew important information about medical history Doctor respected what you had to say Doctor spent enough time with you Courteous and helpful office staff Clerks and receptionists at this doctor’s office were as helpful as you thought they should be Clerks and receptionists at this doctor’s office treated you with courtesy and respect Global rating Doctor rating from 0 to 10

The table below shows the topics that are addressed by the supplemental items. Each topic includes several individual question items that are all related to the same theme.

Topics Addressed by CAHPS Clinician & Group Survey Supplemental Items Adult Primary Care Questionnaire 1.0 • Addressing health literacy • After hours email • Being kept informed about appointment start • Cost of care (prescriptions) • Cost of care (tests) • Doctor role • Doctor thoroughness • Health improvement • Health promotion and education • Help with problems or concerns • Other doctors and providers at your doctor’s office • Provider communication • Provider knowledge of specialist care • Recommend doctor • Shared decisionmaking • Wait time for urgent care • Your care from specialists in the last 12 months • Your most recent visit

Overview of the CAHPS® Clinician & Group Survey

3 of 6

Topics Addressed by CAHPS Clinician & Group Survey Supplemental Items Adult Specialty Care Questionnaire 1.0 • Care you got from this doctor • Coordinating your care • Cost of care (prescriptions) • Doctor role • Shared decisionmaking • Surgery or procedures done by this doctor Child Primary Care Questionnaire 1.0 and 2.0 (beta) • After hours care • Behavioral health • Screening items for children with chronic conditions • Doctor communication • Doctor communication with child • Doctor thoroughness • Health improvement • Identification of site of visit • Prescription medicines • Provider knowledge of specialist care • Shared decisionmaking

Additional Supplemental Items In Development In addition to the topics listed above, there are other CAHPS supplemental items in development that may be of interest to users.

• CAHPS Item Set for Addressing Cultural Competency • CAHPS Item Set for Health Information Technology

More Information A Survey and Reporting Kit for the CAHPS Clinician & Group Survey explains how to prepare and field a questionnaire, analyze the results, and produce consumer-friendly reports. The Kit includes:

• Survey instruments • Protocols and related guidance • Sample documents to help administer the survey

• Analysis programs • Instructions for using analysis programs • Reporting measures

The Kit can be downloaded from the CAHPS Web site at https://www.cahps.ahrq.gov/cahpskit/CG/CGChooseQX.asp.

In addition to the Kit, free technical assistance and other resources are available from the CAHPS User Network, which is sponsored by the U.S. Agency for Healthcare Research and Quality.

• Contact the Help Line at cahps1@ahrq.gov or 1-800-492-9261. • Visit the CAHPS Web site at www.cahps.ahrq.gov.

Overview of the CAHPS® Clinician & Group Survey

4 of 6

Appendix A: Mapping CAHPS Survey Items to the Domains of the Medical Home Core and supplemental CAHPS survey items can be combined into a questionnaire that measures the medical home domains as identified in the Joint Statement on Medical Home issued by the American Academy of Family Physicians (AAFP), the American Academy of Pediatrics (AAP), the American College of Physicians (ACP), and the American Osteopathic Association (AOA). The Joint Statement can be found at http://www.medicalhomeinfo.org/joint%20Statement.pdf. The table below lists the supplemental items that could be added to the core items in the Clinician & Group Survey to address the domains of the medical home.

CAHPS Adult Primary Care Questionnaire

Domains and Supplemental Items*

Doctor communication Doctor interrupted when you were talking Doctor talked too fast when talking with you Doctor used medical words you did not understand Doctor answered all your questions to your satisfaction Doctor gave you complete and accurate information about tests Doctor gave you complete and accurate information about treatment Doctor gave you complete and accurate information about medications Doctor explained what to do if illness/health condition got worse or came back Doctor gave easy to understand instructions about how to take medicines Whole person orientation Doctor really cared about you as a person Doctor understood how health problems affect your day-to-day life Coordination of care Doctor’s office followed up to give you results of blood test, x-ray, or other test (core survey item) Doctor seemed informed and up-to-date about care you received from specialists Health plan, doctor’s office, or clinic helped you to coordinate your care among these doctors or other health providers Shared decisionmaking Doctor talked with you about the pros and cons of each choice for treatment or health care Doctor asked which choice you thought was best for you Involved as much as you wanted in health care decisions Ease of getting doctor to agree with you on the best way to manage your health conditions or problems Your preference for your doctor asking your opinion about the choices you have Your preference for leaving decisions about your treatment or medical care to your doctor Doctor gave your complete and accurate information about choices for your care Chronic disease management Doctor gave you complete and accurate information about plans for your care Doctor gave you complete and accurate information about plans for your follow-up care Doctor asked you to describe how you were going to follow instructions Doctor asked whether you would have problems taking care of this illness or health condition

Overview of the CAHPS® Clinician & Group Survey

5 of 6

CAHPS Adult Primary Care Questionnaire

Domains and Supplemental Items*

Health promotion You and doctor talked about specific things you could do to prevent illness You and doctor talked about a healthy diet and healthy eating habits You and doctor talked about the exercise and physical activity you get You and doctor talked about things in your life that worry you or cause you stress Doctor asked you whether there was a period of time when you felt sad, empty or depressed Doctor gave you all the information you wanted about your health * Includes supplemental items from the Clinician & Group Questionnaire, Health Plan Questionnaire (adapted), and items sets

in development.

Overview of the CAHPS® Clinician & Group Survey

6 of 6

Appendix B: Variations Within the CAHPS Clinician & Group Survey In addition to accounting for differences in age (adult and child) and setting (primary care and specialty care), the Clinician & Group Survey instruments vary in two other ways:

Timeframe: 12-Month Versus Visit The CAHPS Clinician & Group Survey was initially developed to ask patients about their experiences in the last 12 months. Based on feedback from users of the surveys, the CAHPS Team has developed a questionnaire that measures the experiences of patients during a single visit rather than over a period of time. The Visit Questionnaire combines questions about communication and office staff interactions at the most recent visit with questions about access over the last 12 months.

Clinician & Group Survey Instruments Timeframe for Measurement Adult Primary Care Questionnaire 1.0 • Experiences in the last 12 months

Adult Specialty Care Questionnaire 1.0 • Experiences in the last 12 months

Adult Visit Questionnaire (beta)

• Access to care in the last 12 months • Communication during the most recent visit • Office staff interactions during the most

recent visit

Child Primary Care Questionnaire 1.0 • Experiences in the last 12 months

Child Primary Care Questionnaire 2.0 (beta) • Experiences in the last 12 months

Response Scales: 4-Point Versus 6-Point For many of the survey items, the response scales measure the frequency of various experiences. Survey users can choose between two response scales:

• A four-point response scale (Never, Sometimes, Usually, Always) or • A six-point response scale (adds Almost Never and Almost Always to the four-point scale).

The response options for the items about the most recent visit are an expanded Yes/No scale (Yes, Definitely; Yes, Somewhat; No).

Consumer Assessment of Healthcare Providers and Systems

Developing a CAHPS® Clinician & Group Survey to Measure the Medical HomeThe patient-centered medical home (PCMH) is a model for delivering primary care that is patient–centered, comprehensive, coordinated, accessible, and continuously improved through a systems-based approach to quality and safety. The PCMH applies to care for adults, children, and adolescents. Learn more about PCMH at http://www.pcmh.ahrq.gov.

As this model is adopted across the country, many health care organiza-tions are investigating its impact on the effectiveness, efficiency, and patient-centeredness of care. To that end, there is a growing interest in administering a standardized survey that could be used to assess patients’ experiences in practices serving as medical homes. This brief discusses the development of the CAHPS Clinician & Group PCMH Survey to meet this need.

The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) surveys are free, non-proprietary instruments designed to support standard-ized measurement of the experiences of patients with care in a variety of settings. These survey instruments are developed and maintained by a team of prominent research organizations under the auspices of the U.S. Agency for Healthcare Research and Quality (AHRQ). Learn more at https://www.cahps.ahrq.gov.

The CAHPS Clinician & Group Survey, endorsed by the National Quality Forum in July 2007, is comprised of several instruments that enable users to assess and report on the experiences of adults and children in primary and specialty care settings.

As the medical home model is

adopted, many health care

organizations are investigating

its impact.

Development Process for the CAHPS PCMH SurveyThe CAHPS Team has been working with the National Committee for Quality Assurance (NCQA) to develop both adult and child versions of the PCMH Survey. This survey will be used to assess performance related to the NCQA standards for the Physician Practice Connections® - Patient-Centered Medical HomeTM (PPC-PCMH) program. Many of the current PCMH demonstrations and

implementations use the PPC-PCMH program standards. Information on the PPC-PCMH program is available at http://www.ncqa.org/tabid/631/Default.aspx.

As with all other CAHPS surveys, the PCMH Survey is undergoing a rigorous development process that includes:

• Literature review: The CAHPS Team conducted a literature review to ensure that the survey captured the best research in this area.

• Technical Expert Panel input: The CAHPS Team assembled a panel of PCMH experts representing various stakeholders, including providers, health plans, payers,

professional organizations, and regional collaboratives, to provide input on the development and use of a CAHPS survey to assess patient experience with the medical home. Panel members were interviewed in late 2009 and early 2010; an in-person meeting was held in April 2010.

• Stakeholderinput:Stakeholder input is critical to the CAHPS survey development process. NCQA has gathered extensive feedback from stakeholders on the development of a patient experience survey that will be part of the PPC-PCMH program standards.

• Focusgroupinput: Adult patients and parents of children receiving care in medical home practices provided input in summer 2010 to:

♦ Confirm the domains of interest identified by the Technical Expert Panel and other stakeholders;

♦ Identify additional domains, if any; and

♦ Convey how they describe the care they receive in their medical homes.

• Cognitivetesting(EnglishandSpanish): In August 2010, the CAHPS Team conducted cognitive testing of draft PCMH questionnaires for adults and children in both English and Spanish. The draft PCMH questionnaires will be revised as needed based on the findings from the cognitive interviews.This testing version will be available in Fall 2010.

• Fieldtesting: NCQA will conduct a field test of the instru-ment in late 2010. The CAHPS Team is planning to do further field testing to inform implementation issues.

• Psychometricanalysis: The data collected during field test-ing will be analyzed to determine the psychometric proper-ties of the survey items. This analysis will inform the final version of the survey instruments.

• Publicrelease: A final version of the survey will be released in 2011. It will be available on the CAHPS Web site and as part of the NCQA’s updated specifications for the PCMH recognition program.

Organizations interested in testing the PCMH Survey may contact the CAHPS User Network at cahps1@ahrq.gov for more information.

The PCMH Survey is undergoing a rigorous development process.

Key Characteristics of the PCMH SurveyThe PCMH Survey builds on the core items in the CAHPS Clinician & Group Survey.

Every CAHPS survey is composed of core items and supplemental items. The purpose of the required core items is to allow comparisons across provider entities of interest (e.g., medical groups or physicians). Supplemental items offer users a variety of questions that they can add to the core items in order to address specific topics of interest.

The PCMH Survey begins with the CAHPS Clinician & Group Survey core questionnaire then adds supplemental items to address the PCMH domains. The supplemental items derive from both existing CAHPS surveys as well as other surveys that were identified through the literature review or by the Technical Expert Panel and other stakeholders. Supplemental items drawn from other surveys are often amended to ensure that they are consistent with the design principles for CAHPS surveys.

Consumer Assessment of Healthcare Providers and Systems

The PCMH Survey asks about experiences with providers and staff in the office.

Unlike other Clinician & Group Surveys that ask about “this doctor,” the CAHPS PCMH Survey asks patients about their experiences interacting with three different accountable entities in the practice:

• “Thisprovider”(anindividualcliniciandefinedinthefirstquestion)

• “Careteam”(allthepeoplewhoworkwithyourprovidertogive you health care)

• Clerksandreceptionistsatthisprovider’soffice

The PCMH Survey asks about experiences over the last 12 months.

The CAHPS PCMH Survey asks about care received in the last 12 months rather than just a single visit. The majority of stakeholders providing input into the development process felt that a 12-month timeframe was more appropriate than focusing on a particular visit because the medical home concepts do not necessarily occur at specific visits but represent care that is received over time and between visits.

The PCMH Survey covers many aspects of patient-centered care.

The PCMH Survey focuses on the following domains:

• Access

• Communication

• Coordination

♦ Care from other providers

♦ Care from others on the care team

• Comprehensiveness

• Shareddecisionmaking

• Wholepersonorientation

• Self-managementsupport

♦ Chronic disease management

♦ Health promotion

The final number of survey items needed to cover all of these topics has not yet been determined. However, the length of the survey is not expected to pose a problem. The CAHPS

Team is aware that providers accustomed to obtaining patient input through just a few questions or a brief comment card are concerned that patients may not complete a survey that seems lengthy. However, research has confirmed that survey length does not negatively affect response rates even for surveys with over 75 questions.

The PCMH Survey can be administered by mail, telephone, or both.

CAHPS surveys are typically administered through mail, telephone, or a mixed mode of mail with telephone follow-up. These modes are recommended by the CAHPS Team because they have been proven to achieve comparable survey results. The CAHPS Team is currently investigating the feasibility and impact of using the Web to administer the survey.

Some providers currently administer their patient surveys byhandingthesurveytothepatientduringanofficevisit.Survey researchers have found that data obtained through in-officeadministrationisnotcomparabletothatcollectedbymail and telephone administration.

Additional CAHPS Survey ResourcesSeveral free resources to support CAHPS surveys are available from the CAHPS User Network, which is sponsored by the U.S. Agency for Healthcare Research and Quality.

CAHPS Clinician & Group Survey and Reporting Kit

The CAHPS Clinician & Group Survey is part of a Survey and Reporting Kit that explains how to prepare and field a CAHPS questionnaire, analyze the results, and produce consumer-friendly reports. Many of the resources in the current Kit can be used for the PCMH Survey. Once the PCMH Survey is finalized, it will also be integrated into the Kit.The Kit includes:

• Finalsurveyinstruments

• Datacollectionprotocolsandrelatedguidance

• Sampledocumentstohelpadministerthesurvey,suchasexamples of notification and reminder letters

• Analysisprograms

• Instructionsforusinganalysisprograms

• Reportingmeasures

The Kit can be downloaded from the CAHPS Web site at https://www.cahps.ahrq.gov/cahpskit/CG/CGChooseQX.asp.

CAHPS Database

The CAHPS Database is the national repository for data from the CAHPS Health Plan and Clinician & Group Surveys. The CAHPS Clinician & Group portion of the Comparative Database is currently under development. Comparative data for the CAHPS Clinician & Group Survey will be made available as survey sponsors implement the CAHPS Clinician & Group Surveys and submit data to the CAHPS Database. Current plans include integrating the PCMH Survey into the CAHPS Database once the survey is finalized and implemented.

More information on the CAHPS Database can be found at https://www.cahps.ahrq.gov/content/ncbd/ncbd_Intro.asp.

To remain up-to-date about the CAHPS Patient-Centered Medical Home Survey, visit https://www.cahps.ahrq.gov/content/products/CG/PROD_CG_PCMH.asp

Consumer Assessment of Healthcare Providers and Systems

For free technical assistance and other resources:

• ContacttheHelpLineatcahps 1@ahrq.gov or 1-800-492-9261.

• VisittheCAHPSWebsiteat https://www.cahps.ahrq.gov.

AHRQ Pub. No. 10-CAHPS003 September 201030980.0910.8486010160

Health Affairs 

May 2010 

 

"Measuring Patient Experience As A Strategy For Improving Primary Care" 

Katherine Browne, Deborah Roseman, Dale Shaller and Susan Edgman‐Levitan 

 

Abstract:  

Patients value the interpersonal aspects of their health care experiences. However, faced with multiple resource demands, primary care practices may question the value of collecting and acting upon survey data that measure patients’ experiences of care. The Consumer Assessment of Healthcare Providers and Systems (CAHPS) suite of surveys and quality improvement tools supports the systematic collection of data on patient experience. Collecting and reporting CAHPS data can improve patients’ experiences, along with producing tangible benefits to primary care practices and the health care system. We also argue that the use of patient experience information can be an important strategy for transforming practices as well as to drive overall system transformation.   

http://content.healthaffairs.org/content/29/5/921.abstract