6
Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners Robert H. Lohr, MD; Colin P. West, MD, PhD; Margaret Beliveau, MD; Paul R. Daniels, MD; Mark A. Nyman, MD; William C. Mundell, MD; Nina M. Schwenk, MD; Jayawant N. Mandrekar, PhD; James M. Naessens, ScD; and Thomas J. Beckman, MD Abstract Objective: To compare the quality of referrals of patients with complex medical problems from nurse practitioners (NPs), physician assistants (PAs), and physicians to general internists. Patients and Methods: We conducted a retrospective comparison study involving regional referrals to an academic medical center from January 1, 2009, through December 31, 2010. All 160 patients referred by NPs and PAs combined and a random sample of 160 patients referred by physicians were studied. Five experienced physicians blinded to the source of referral used a 7-item instrument to assess the quality of referrals. Internal consistency, interrater reliability, and dimensionality of item scores were determined. Differences between item scores for patients referred by physicians and those for patients referred by NPs and PAs combined were analyzed by using multivariate ordinal logistical regression adjusted for patient age, sex, distance of the referral source from Mayo Clinic, and Charlson Index. Results: Factor analysis revealed a 1-dimensional measure of the quality of patient referrals. Interrater reliability (intraclass correlation coefcient for individual items: range, 0.77-0.93; overall, 0.92) and in- ternal consistency for items combined (Cronbach a¼0.75) were excellent. Referrals from physicians were scored higher (percentage of agree/strongly agree responses) than were referrals from NPs and PAs for each of the following items: referral question clearly articulated (86.3% vs 76.0%; P¼.0007), clinical infor- mation provided (72.6% vs 54.1%; P¼.003), documented understanding of the patients pathophysiology (51.0% vs 30.3%; P<.0001), appropriate evaluation performed locally (60.3% vs 39.0%; P<.0001), appropriate management performed locally (53.5% vs 24.1%; P<.0001), and condence returning patient to referring health care professional (67.8% vs 41.4%; P<.0001). Referrals from physicians were also less likely to be evaluated as having been unnecessary (30.1% vs 56.2%; P<.0001). Conclusion: The quality of referrals to an academic medical center was higher for physicians than for NPs and PAs regarding the clarity of the referral question, understanding of pathophysiology, and adequate prereferral evaluation and documentation. ª 2013 Mayo Foundation for Medical Education and Research n Mayo Clin Proc. 2013;88(11):1266-1271 P rimary care workforce shortages are pre- dicted for many areas of the United States in the next decade. 1,2 Nurse prac- titioners (NPs) and physician assistants (PAs) are increasingly being used to improve health care access. For example, new models of care delivery, such as the Patient Centered Medical Home, often include NPs and PAs along with primary care physicians. 3 All states require that PAs be supervised by physicians, and many states require that NPs be supervised by physicians. However, the availability of supervision for NPs and PAs varies, and several states allow NPs to practice independently. 4-6 Indeed, allowing NPs to practice independently in all states has been identied as a potential solution to the pri- mary care workforce shortage. 7,8 However, controversy exists regarding the roles of physi- cians and NPs within health care teams and levels of supervision. 9 It has been suggested that NPs and PAs can perform the roles of primary care physi- cians. 10-12 Although patients with multiple From the Division of General Internal Medicine (R.H.L., C.P.W., M.B., P.R.D., M.A.N., W.C.M., N.M.S., T.J.B.), Divi- sion of Biomedical Statistics and Informatics (J.N.M.), and Division of Health Care Policy and Research (J.M.N.), Mayo Clinic, Rochester, MN. 1266 Mayo Clin Proc. n November 2013;88(11):1266-1271 n http://dx.doi.org/10.1016/j.mayocp.2013.08.013 www.mayoclinicproceedings.org n ª 2013 Mayo Foundation for Medical Education and Research ORIGINAL ARTICLE

Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners

Embed Size (px)

Citation preview

Page 1: Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners

ORIGINAL ARTICLE

From the Division of GInternal Medicine (R.H.LC.P.W., M.B., P.R.D., M.W.C.M., N.M.S., T.J.B.),sion of Biomedical Statiand Informatics (J.N.M.)Division of Health Careand Research (J.M.N.), MClinic, Rochester, MN.

1266

Comparison of the Quality of Patient ReferralsFrom Physicians, Physician Assistants, and Nurse

Practitioners

eneral.,A.N.,Divi-stics, andPolicyayo

Robert H. Lohr, MD; Colin P. West, MD, PhD; Margaret Beliveau, MD;Paul R. Daniels, MD; Mark A. Nyman, MD; William C. Mundell, MD;

Nina M. Schwenk, MD; Jayawant N. Mandrekar, PhD; James M. Naessens, ScD;and Thomas J. Beckman, MD

Abstract

Objective: To compare the quality of referrals of patients with complex medical problems from nursepractitioners (NPs), physician assistants (PAs), and physicians to general internists.Patients and Methods: We conducted a retrospective comparison study involving regional referrals to anacademic medical center from January 1, 2009, through December 31, 2010. All 160 patients referred byNPs and PAs combined and a random sample of 160 patients referred by physicians were studied. Fiveexperienced physicians blinded to the source of referral used a 7-item instrument to assess the quality ofreferrals. Internal consistency, interrater reliability, and dimensionality of item scores were determined.Differences between item scores for patients referred by physicians and those for patients referred by NPsand PAs combined were analyzed by using multivariate ordinal logistical regression adjusted for patientage, sex, distance of the referral source from Mayo Clinic, and Charlson Index.Results: Factor analysis revealed a 1-dimensional measure of the quality of patient referrals. Interraterreliability (intraclass correlation coefficient for individual items: range, 0.77-0.93; overall, 0.92) and in-ternal consistency for items combined (Cronbach a¼0.75) were excellent. Referrals from physicians werescored higher (percentage of agree/strongly agree responses) than were referrals from NPs and PAs for eachof the following items: referral question clearly articulated (86.3% vs 76.0%; P¼.0007), clinical infor-mation provided (72.6% vs 54.1%; P¼.003), documented understanding of the patient’s pathophysiology(51.0% vs 30.3%; P<.0001), appropriate evaluation performed locally (60.3% vs 39.0%; P<.0001),appropriate management performed locally (53.5% vs 24.1%; P<.0001), and confidence returning patientto referring health care professional (67.8% vs 41.4%; P<.0001). Referrals from physicians were also lesslikely to be evaluated as having been unnecessary (30.1% vs 56.2%; P<.0001).Conclusion: The quality of referrals to an academic medical center was higher for physicians than for NPsand PAs regarding the clarity of the referral question, understanding of pathophysiology, and adequateprereferral evaluation and documentation.

ª 2013 Mayo Foundation for Medical Education and Research n Mayo Clin Proc. 2013;88(11):1266-1271

P rimary care workforce shortages are pre-dicted for many areas of the UnitedStates in the next decade.1,2 Nurse prac-

titioners (NPs) and physician assistants (PAs)are increasingly being used to improve healthcare access. For example, new models of caredelivery, such as the Patient Centered MedicalHome, often include NPs and PAs along withprimary care physicians.3

All states require that PAs be supervised byphysicians, and many states require that NPsbe supervised by physicians. However, the

Mayo Clin Proc. n November 2013;88(www.mayoclinicproceedings.org n

availability of supervision for NPs and PAsvaries, and several states allow NPs to practiceindependently.4-6 Indeed, allowing NPs topractice independently in all states has beenidentified as a potential solution to the pri-mary care workforce shortage.7,8 However,controversy exists regarding the roles of physi-cians and NPs within health care teams andlevels of supervision.9

It has been suggested that NPs and PAscan perform the roles of primary care physi-cians.10-12 Although patients with multiple

11):1266-1271 n http://dx.doi.org/10.1016/j.mayocp.2013.08.013ª 2013 Mayo Foundation for Medical Education and Research

Page 2: Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners

NP/PA VS PHYSICIAN REFERRALS

comorbidities in the intensive care unit have beenmanaged by NPs and PAs with the onsite avail-ability of a critical care physician,13-15 there hasbeen little research on the abilities of NPs andPAs to independently manage patients with com-plex medical problems. Research indicates thatprimary care physicians have difficulty caringfor patients with multiple interacting diseases,which implies that NPs and PAsmight also strug-gle with managing such patients.16,17 General in-ternists are trained to manage community-basedpatients who are referred with undifferentiatedproblems and multiple comorbidities.18 Thefact that patients are referred defines them ascomplex in the eyes of the referring health careprofessional. As the need for primary care ser-vices increases, it will be important to determinewhether the care of patients with complex med-ical problems requires the expertise of physicianseither locally or in a consultative referral practice.For example, discussions among supervisingphysicians, NPs, and PAs locally might suffi-ciently clarify a patient’s medical issues, and soa referral would be unnecessary.

The objective of this study was to deter-mine whether differences exist for referrals toa large midwestern academic practice amongphysicians, NPs, and PAs on the basis of (1)experienced physicians’ validated blinded as-sessments of the quality of previous evaluationand referrals and (2) patient complexity ascalculated by using the Charlson Index.19

PATIENTS AND METHODS

Study Design, Setting, and ParticipantsThe Division of General Internal Medicine,Department of Internal Medicine, at the MayoClinic in Rochester, Minnesota, is an academicpractice that specializes in consultative internalmedicine and provides little ongoing primarycare. Patients are referred to the Division of Gen-eral Internal Medicine from throughout theworld, but most live within 500 miles of MayoClinic and are referred by primary care physi-cians, NPs, and PAs who practice in rural com-munities and small cities. Accordingly, theDivision of General Internal Medicine offers aunique opportunity to study the characteristicsof referrals from primary care practices for assis-tance with evaluating and managing patientswith complex, undifferentiated problems.

Mayo Clin Proc. n November 2013;88(11):1266-1271 n http://dx.dowww.mayoclinicproceedings.org

We conducted a retrospective comparisonstudy of all 160 patients referred by NPs andPAs, as well as a computer-generated randomsample of 160 patients referred by physicians,to the Division of General Internal Medicinefrom January 1, 2009, through December 31,2010. Data were de-identified so that investiga-tors reviewing the patient histories would beblinded to the referral source and patient identi-ties.We estimated that this 2-sample comparisonof 160 participants per group would provide80% power to detect a small to moderateCohen’s d effect size of 0.32.

During the time frame of our study, approx-imately 800 regional patients per year werereferred to the Division of General Internal Med-icine, with 10% of these referrals coming fromNPs and PAs. For this study, 160 patients werereferred by 90 NPs or PAs. Eight of these patientswere excluded because of missing authorizationto use medical records for research purposes,and 6 were excluded because of incompletede-identification. In addition, we identified arandom sample of 160 of the remaining patientswho were referred by 126 individual physicians.Five of these patients were excluded because ofmissing authorization to use the medical recordsfor research purposes, and 9 were excludedbecause of incomplete de-identification, whichresulted in a total of 292 patient referrals avail-able for analysis, with 146 referrals in eachcohort. This study was approved by the MayoClinic Institutional Review Board.

To better understand the level of physiciansupervision that NPs and PAs receive, we maileda separate survey to every NP and PA for whomwe had current contact information (n¼88) whohad referred a patient to the Division of GeneralInternal Medicine during the study period. Thesurvey responses of NPs and PAs were anony-mous; therefore, we were unable to link the sur-vey and comparison data. The single surveyquestion we asked was: “How often do youdiscuss patients with physician colleagues beforereferring your patient to Mayo Clinic?” Theresponse options to this questionwere as follows:never, some of the time, about half of the time,most of the time, and always.

Study VariablesAssessed patient characteristics included pa-tient age, sex, and distance of the referralsource from Mayo Clinic. Diseases for each

i.org/10.1016/j.mayocp.2013.08.013 1267

Page 3: Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners

MAYO CLINIC PROCEEDINGS

1268

patient at the time of referral were used to calcu-late Charlson Index scores. Physician attitudesregarding the quality and appropriateness of re-ferrals were recorded for the following 7 items(on a Likert scale: 1 ¼ strongly disagree, 2 ¼disagree, 3 ¼ neither agree nor disagree, 4 ¼agree, and 5 ¼ strongly agree): (1) the clinicalquestion was clearly articulated, (2) the clinicalinformation was provided, (3) the clinical ques-tion reflected understanding of pathophysi-ology, (4) the patient had been appropriatelyevaluated locally, (5) the patient had beenappropriately managed locally, (6) the clinicalquestion could have been handled by a primarycare health care professional locally, and (7) I amconfident sending the patient back to the refer-ring health care professional. The validation ofthe scores from these 7 items is described inthe next section.

Instrument Development and ValidationAn instrument was designed to measure physi-cians’ attitudes about the quality of patientreferrals to the Division of General InternalMedicine. Instrument items addressed whetherpatients were appropriately referred (items 1-3in the preceding paragraph) and whether pa-tients were being appropriately managed bytheir local health care professionals (items 4-7in the preceding paragraph). The content valid-ity for these items was determined with inputfrom one of the authors (T.J.B.), who has expe-rience in scale development and validation, anda panel of 8 Mayo Clinic physicians with a min-imum of 10 years of experience in the generalinternal medicine practice, who analyzed theitems for clarity and appropriateness and sug-gested no revisions.

The instrument was pilot tested on 5 pa-tient records by all physician authors. Theinterrater reliability across 8 physician ratersfor each item and overall was determinedfrom intraclass correlation coefficients, whichwere interpreted as follows: less than 0.4 ¼poor, 0.4 to 0.75 ¼ fair to good, and greaterthan 0.75 ¼ excellent.20 Results of this interimanalysis revealed an excellent interrater reli-ability of the intraclass correlation coefficientacross all items (range, 0.77-0.93) and overall(0.92). On the basis of this analysis, it wasdetermined that the physician raters had aconsistent understanding of the rating instru-ment and its application to the patient records.

Mayo Clin Proc. n November 2013;88(

Five of these physicians then used the in-strument to assess the quality of patient referralsto Mayo Clinic. The physician raters wereblinded to the source of referral (physician ornonphysician). The factor analysis of the physi-cian assessments was used to determine thedimensionality of the item scores. To accountfor the clustering of multiple ratings within raterand referral source (physician vs NP or PA)groupings, we generated an adjusted correlationmatrix by using generalized estimating equa-tions. This adjusted correlation matrix wasthen used to perform factor analysis. In addition,for a sensitivity analysis, we performed a factoranalysis by using an unadjusted correlation ma-trix and within rater and referral source combi-nations separately. Factors were extracted byusing the eigenvalue method. Internal consis-tency for each factor, as well as overall consis-tency, was calculated by using Cronbach a,and a>0.7 was considered acceptable.

Statistical AnalysesPatient characteristics were reported by usingstandard descriptive statistics. Differences be-tween item scores for patients referred by physi-cians compared with those referred by PAs andNPs were assessed by using multivariate ordinallogistical regression, adjusting for patient age,sex, distance of the referral source from MayoClinic, and Charlson Index. Differences betweengroups for Charlson Index scores were evaluatedby using the analysis of variance. Two-tailed sta-tistical significance was set at a¼.05. Statisticalanalyses were conducted by using SAS version9.1 (SAS Institute Inc).

RESULTSThe characteristics of patients referred to theDivision of General Internal Medicine are sum-marized in Table 1. Patients referred by physi-cians were older than those referred by NPsandPAs (P¼.04). Patients referred byphysicianswere also more likely to be men (P¼.0003).Therewere no statistically significant differencesbetween groups in terms of the distance traveled(P¼.71) or disease complexity (determinedfrom the Charlson Index, P¼.1). The reasonsfor referral included issues relating to all internalmedicine subspecialties, as well as undifferenti-ated symptoms such as fatigue and weight loss.There were no substantial differences betweenstudy groups in terms of the reasons for referral.

11):1266-1271 n http://dx.doi.org/10.1016/j.mayocp.2013.08.013www.mayoclinicproceedings.org

Page 4: Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners

TABLE 1. Characteristics of Regional Patients Referred by Physicians andNonphysicians to the Division of General Internal Medicine at Mayo Clinica

CharacteristicPhysicians

(n¼146 patients)Nonphysicians

(n¼146 patients) P value

Age (y) .0415-34 15 (10.3) 21 (14.4)35-49 25 (17.1) 39 (26.7)50-64 41 (28.1) 45 (30.8)65-79 45 (30.8) 26 (17.8)�80 20 (13.7) 15 (10.3)

Sex .0003Male 62 (42.5) 32 (21.9)Female 84 (57.5) 114 (78.1)

Distance (mi) .71<120 128 (87.7) 131 (89.7)�120 18 (12.3) 15 (10.3)

Charlson Index scoreb 0.97�1.50 0.62�1.09 .10

aData are presented as No. (percentage) of patients or mean þ/� SD.bCharlson Index is a comorbidity score giving larger weights to conditions related to lower 1-yearsurvival.

NP/PA VS PHYSICIAN REFERRALS

The factor analysis revealed a 1-dimensionalmeasure of the quality of patient referrals fromphysicians, PAs, and NPs. Internal consistencyfor all items combined was excellent (Cronbacha¼0.75). Referrals from physicians were scoredsignificantly higher (percentage of agree/stronglyagree responses) thanwere referrals fromPAs andNPs for each of the following instrument items:referral question clearly articulated (86.3% vs76.0%; P¼.0007), clinical information provided(72.6% vs 54.1%; P¼.003), documented under-standing of the patient’s pathophysiology (51.0%vs 30.3%; P<.0001), appropriate evaluation per-formed locally (60.3% vs 39.0%; P<.0001),appropriate management performed locally(53.5% vs 24.1%; P<.0001), and confidencereturning patient to referring health care profes-sional (67.8% vs 41.1%; P<.0001). Physician re-ferrals were also less likely to be evaluated asadding little clinical value (30.1% vs 56.2%;P<.0001) (Table 2).

Regarding the survey of all referring NPsand PAs over the study time frame, 44 NPs/PAs responded to 88 mailed letters (responserate 50%). Twenty-six of 44 respondents re-ported that either they “never” discussedreferral plans with their supervising physiciansor they did so “some of the time,” and only 3of 44 respondents reported that they “always”discussed referral plans with their supervisingphysicians.

DISCUSSIONTo our knowledge, this is the first study toreport that the quality of patient referralsfrom primary care health care professionals togeneral internists at an academic medical cen-ter, as determined by a panel of experienced in-ternal medicine faculty members blinded tosource of referral, is substantially higher forphysicians than for PAs and NPs with respectto several characteristics including clarity ofthe referral question, understanding of patho-physiology, and adequate prereferral evalua-tion and documentation. These preliminaryfindings suggest that there is an opportunityto improve the quality of all patient referralsfrom primary care practices, but especiallythose that use NPs and PAs, by involving inte-grated health care teams that combine the skillsof physicians, NPs, and PAs.

The reasons why the quality of referrals dif-fers between physicians and nonphysicians are

Mayo Clin Proc. n November 2013;88(11):1266-1271 n http://dx.dowww.mayoclinicproceedings.org

likely multifactorial and might best be viewedin the context of team-based care and the broaderhealth care system. Specifically, these differencescould be considered with respect to interactingpatient, health care professional. and system-related factors. Patients who require referral toa tertiary medical center are typically more com-plex and undifferentiated in terms of a diagnosis.Although there is evidence that NPs and PAs candeliver effective primary care,10-12 there is littleresearch on their abilities to independently man-age patients with undifferentiated and complexproblems. However, there are many examplesof multidisciplinary teams including NPs, PAs,and physicians that provide excellent care topatients with complex medical problems. Suchmodels have been found in the intensive careunit,13-15 trauma unit,21 emergency depart-ment,22 and hospital ward.23Notably, in all thesesettings, NPs and PAs had immediate access tophysician support, whereas this level of supportis not necessarily available in all outpatient prac-tice settings.11 Indeed, our survey of referringNPs and PAs indicated that they usually didnot consult with a physician colleague beforereferring a patient. Potential reasons for limitedsupervision in outpatient practices would in-clude high patient volumes, abbreviated appoint-ment times, and geographic separation betweenphysicians, NPs, and PAs.

All states require a supervisory relationshipbetween physicians and PAs4,10; however, this

i.org/10.1016/j.mayocp.2013.08.013 1269

Page 5: Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners

TABLE 2. Comparison of the Quality of Patient Referrals From Physicians and Nonphysicians to the Division of General Internal Medicine atMayo Clinic

Item

Agree or strongly agree (%)

P valueaPhysicians

(n¼146 patients)Nonphysicians

(n¼146 patients)

1. The clinical question was clearly articulated 86.3 76.0 .00072. The clinical information was provided 72.6 54.1 .0033. The clinical question reflected understanding of pathophysiology 51.0 30.3 <.00014. The patient had been appropriately evaluated locally 60.3 39.0 <.00015. The patient had been appropriately managed locally 53.5 24.1 <.00016. The clinical question could have been handled by a primary care professional locally 30.1 56.2 <.00017. I am confident sending the patient back to the referring professional 67.8 41.1 <.0001

aCalculated by using multivariate ordinal logistic regression analyses adjusted for patient age, sex, distance of the referral source from Mayo Clinic, and Charlson Index.

MAYO CLINIC PROCEEDINGS

1270

is not true for NPs, who are allowed to practiceindependently in many states.5,6 Therefore, theactual level of supervision is not known. The de-gree of NP and PA supervision is more clearly ar-ticulated in health maintenance organizationsand multispecialty clinics,11 as well as in aca-demic medical centers, in which NPs and PAsare increasingly used to offset resident dutyhour limitations.23 It appears that guidelinesregarding NP and PA supervision are generallydetermined locally.We could find little literaturethat addresses the situation of NP and PA super-vision,11 suggesting the potential need for moreuniversal guidelines regarding best practices forcollaboration between NPs, PAs, and physicians.Therefore, future research should determinewhether physician supervision of NPs and PAshas an effect on the quality of patient evaluationand referral to academic medical centers.

Our assessment of the quality of referralsto an academic general internal medicine prac-tice revealed that the overall quality of referralswas suboptimal. Consequently, methods toimprove this process should be studied, withshared input from both the referring healthcare professionals and the referral practices,such as those typically found at academicmedical centers. Although some may arguethat using NPs and PAs reduces health carecosts, our study findings suggest that inappro-priate referrals to tertiary referral centers byNPs/PAs could offset any potential savings.Finally, we postulate that optimizing the qual-ity of referrals from both nonphysicians andphysicians will be necessary to achieve an inte-grated, functional, national health care systemthat uses resources effectively.

Mayo Clin Proc. n November 2013;88(

Our study has several limitations.When con-ducting the assessments and statistical analyses,we pooled referral data for both NPs and PAswhile recognizing that they have training uniqueto their respective degrees and, potentially,different skills. However, we would highlightthat NPs and PAs are often considered to besimilar groups in the contexts of practice settings,health care professional recruitment, and the liter-ature.11 Similarly, we were unable to determinewhat specialties comprised the referring physi-cians in our study cohort, although the internalMayo Clinic referral data indicate that most ofthese physicians are primary care internists andfamily physicians. Many attributes of the referralsin this study are unknown, including the experi-ence level of the referring health care professional,the level of supervision provided to NPs and PAs,patient panel sizes of the local practices, patientvolumes seen on a typical day, and the extent towhich patients may have requested their own re-ferrals to Mayo Clinic. Nonetheless, our study re-sults were based on a multivariate analysis thatadjusted for patient age, sex, distance of thereferral source from Mayo Clinic, and diseasecomplexity. Finally, we would underscore thatthis study reports only the assessments, eventhough based on a validated measure, of experi-enced physicians. Although we identified differ-ences in the quality of referrals from NPs, PAs,and physicians, additional research is needed todeterminewhether differences in higher-level, pa-tient-related outcomes exist.

CONCLUSIONWe found that the quality of previsit care andpatient referrals to general internists at a tertiary

11):1266-1271 n http://dx.doi.org/10.1016/j.mayocp.2013.08.013www.mayoclinicproceedings.org

Page 6: Comparison of the Quality of Patient Referrals From Physicians, Physician Assistants, and Nurse Practitioners

NP/PA VS PHYSICIAN REFERRALS

medical center, on the basis of a validated assess-ment by academic faculty members, was higherfor physicians than for NPs and PAs. Althoughprevious research has examined the value ofcare provided by NPs and PAs in supervisedsettings, we are unaware of previous researchcomparing referrals and care provided by physi-cians, NPs, and PAs. Our findings indicate theneed for future studies to compare patient refer-rals by physicians, NPs, and PAs with respect tohigher-level outcomes such as patient satisfac-tion and quality of care metrics, as well asresearch into optimal interdisciplinary modelsfor teams involving physicians, NPs, and PAs.

Abbreviations and Acronyms: NP = nurse practitioner;PA = physician assistant

Grant Support: ThisworkwasfinanciallysupportedbytheDepart-ment of Internal Medicine, Mayo Clinic, Rochester, MN.

Correspondence: Address to Robert H. Lohr, MD, Divisionof General Internal Medicine, Mayo Clinic, 200 First St SW,Rochester, MN 55905 ([email protected]).

REFERENCES1. MacDowell M, Glasser M, Fitts M, Nielsen K, Hunsaker M.

A national view of rural health workforce issues in the USA.Rural Remote Health. 2010;10(3):1531.

2. Dill MJ, Salsberg ES; Center for Workforce Studies. The Com-plexities of Physician Supply and Demand: Projections Through2025. Washington, DC: American Association of Medical Col-leges; November 2008.

3. American Academy of Family Practice, American Academyof Pediatrics, American College of Physicians, American Osteo-pathic Association. Joint Principles of the Patient Centered MedicalHome. 2007; http://www.acponline.org/acp_policy/policies/joint_principles_pcmh_2007.pdf. Accessed September 19, 2013.

4. American Academy of Physician Assistants website. http://www.aapa.org. Accessed June 13, 2013.

5. American College of Nurse Practitioners website. http://www.acnpweb.org. Accessed June 13, 2013.

6. Making the numbers count. Center for Health Professions, Uni-versity of California at San Francisco website. http://futurehealth.ucsf.edu/Public/Publications-and-Resources/Content.aspx?topic¼Making_the_Numbers_Count. Published September 1, 2012.Accessed June 13, 2013.

Mayo Clin Proc. n November 2013;88(11):1266-1271 n http://dx.dowww.mayoclinicproceedings.org

7. Emanuel E, Tanden N, Altman S, et al. A systemic approach tocontaining health care spending. N Engl J Med. 2012;367(10):949-954.

8. Fairman JA, Rowe JW, Hassmiller S, Shalala DE. Broadeningthe scope of nursing practice. N Engl J Med. 2011;364(3):193-196.

9. Blumenthal D, Abrams MK. Putting aside preconceptionsdtime for dialogue among primary care clinicians. N Engl JMed. 2013;368(20):1933-1934.

10. Atwater A, Bednar S, Hassman D, Khouri J. Nurse practitionersand physician assistants in primary care. Dis Mon. 2008;54(11):728-744.

11. Jacobson PD, Parker LE, Coulter ID. Nurse practitioners andphysician assistants as primary care providers in institutional set-tings. Inquiry. 1998-1999;35(4):432-446.

12. Laurant M, Reeves D, Hermens R, Braspenning J, Grol R,Sibbald B. Substitution of doctors by nurses in primary care.Cochrane Database Syst Rev. 2005;(2):CD001271.

13. Gershengorn HB, Johnson MP, Factor P. The use of nonphysi-cian providers in adult intensive care units. Am J Respir Crit CareMed. 2012;185(6):600-605.

14. Gershengorn HB, Wunsch H, Wahab R, et al. Impact ofnonphysician staffing on outcomes in a medical ICU. Chest.2011;139(6):1347-1353.

15. Kleinpell RM, Ely EW, Grabenkort R. Nurse practitioners andphysician assistants in the intensive care unit: an evidence-based review. Crit Care Med. 2008;36(10):2888-2897.

16. Fried TR, Tinetti ME, Iannone L. Primary care clinicians’ experi-ences with treatment decision making for older persons withmultiple conditions. Arch Intern Med. 2011;171(1):75-80.

17. Subramanian U, Kerr EA, Klamerus ML, Zikmund-Fisher BJ,Holleman RG, Hofer TP. Treatment decisions for complex pa-tients: differences between primary care physicians and midlevelproviders. Am J Manag Care. 2009;15(6):373-380.

18. Larson EB, Fihn SD, Kirk LM, et al. The future of general internalmedicine. Report and recommendations from the Society ofGeneral Internal Medicine (SGIM) Task Force on the Domainof General Internal Medicine. J Gen Intern Med. 2004;19(1):69-77.

19. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A newmethod of classifying prognostic comorbidity in longitudinalstudies: development and validation. J Chronic Dis. 1987;40(5):373-383.

20. Landis JR, Koch GG. The measurement of observer agreementfor categorical data. Biometrics. 1977;33(1):159-174.

21. Gillard JN, Szoke A, Hoff WS, Wainwright GA, Stehly CD,Toedter LJ. Utilization of PAs and NPs at a level I trauma cen-ter: effects on outcomes. JAAPA. 2011;24(7):34, 40-43.

22. Sherwood K, Sugerman S, Bossart P, et al. EDOU staffing byPAs: what are the effects on patient outcomes? JAAPA. 2011;24(8):31-34, 37.

23. Moote M, Krsek C, Kleinpell R, Todd B. Physician assistant andnurse practitioner utilization in academic medical centers. Am JMed Qual. 2011;26(6):452-460.

i.org/10.1016/j.mayocp.2013.08.013 1271