Do new workforce roles reduce waiting times in ED? A difference-in-difference evaluation using hospital administrative data

H

Dda

Aa

b

V

a

ARRA

KWED

1

chcteir

j

0

ARTICLE IN PRESSG ModelEAP-3330; No. of Pages 6

Health Policy xxx (2014) xxx–xxx

Contents lists available at ScienceDirect

Health Policy

journa l h om epa ge: www.elsev ier .com/ locate /hea l thpol

o new workforce roles reduce waiting times in ED? Aifference-in-difference evaluation using hospitaldministrative data

nthony Scotta, Jongsay Yongb,∗

Melbourne Institute of Applied Economic & Social Research, University of Melbourne, Victoria 3010, AustraliaMelbourne Institute of Applied Economic & Social Research, University of Melbourne, L5, 111 Barry Street, FBE Building,ictoria 3010, Australia

r t i c l e i n f o

rticle history:eceived 7 December 2012eceived in revised form 13 October 2014ccepted 12 November 2014

eywords:orkforce innovations

D outcome evaluationifference-in-differences analysis

a b s t r a c t

This paper evaluates the effect of introducing two new workforce roles under a pilot pro-gram conducted in Victoria, Australia. The trial took place at a regional hospital’s emergencydepartment (ED) between 1 July 2008 and 30 June 2009. The evaluation is based on threeoutcome measures: waiting time (in minutes) at ED before treatment; proportion of presen-tations with waiting time on target; and length of stay (in days), for ED presentations thatled to in-patient admissions. The technique of difference-in-differences analysis is used. Atotal of 142,980 patient records from the pilot hospital and three comparison hospitals wereextracted from the Victorian Emergency Minimum Dataset (VEMD). Further, 21,925 recordsof patients whose ED presentations led to in-patient admissions were extracted from theVictorian Admitted Episodes Dataset (VAED). The evaluation finds the piloted roles have
lowered waiting time and raised the proportion of on-target presentations. These effectswere found to be the strongest for less urgent triage categories. However, the evidenceon in-patient length of stay was mixed. The results provide positive evidence that newworkforce roles can be effective in improving the efficiency of emergency care delivery.
© 2014 Published by Elsevier Ireland Ltd.

. Background

New technologies, increasing population size, increasedommunity expectations and the aging of the populationave put enormous pressure on health care in all advancedountries. The demand for emergency care is no excep-ion; overcrowding is a common phenomenon facing many

Please cite this article in press as: Scott A, Yong Jin ED? A difference-in-difference evaluation using hohttp://dx.doi.org/10.1016/j.healthpol.2014.11.018

mergency departments (ED). In the face of demand ris-ng at a rate that places pressure on existing services andesources, raising the efficiency and performance of ED is

∗ Corresponding author. Tel.: +61 3 83442121; fax: +61 3 83442111.E-mail addresses: [email protected] (A. Scott),

[email protected] (J. Yong).

http://dx.doi.org/10.1016/j.healthpol.2014.11.018168-8510/© 2014 Published by Elsevier Ireland Ltd.

becoming a central strategy in addressing ED overcrowd-ing. In view of the many undesirable consequences ofcrowding [1,2], various ED reform and process redesigninitiatives have been proposed and investigated in the lit-erature [3–6].

Innovation in ED workforce redesign has been oneimportant and promising area to explore for improving thedelivery of emergency care; an example is the ChangingWorkforce Program (CWP) in the UK [7]. However, therehave been few evaluations of changes to ED workforce mixand roles [8]. This paper contributes to the literature by

. Do new workforce roles reduce waiting timesspital administrative data. Health Policy (2014),

applying the difference-in-differences method to evaluatethe new workforce roles introduced in a pilot program ata regional hospital (henceforth referred to as Hospital A)in Victoria, Australia. The pilot program was introduced

dx.doi.org/10.1016/j.healthpol.2014.11.018


http://www.sciencedirect.com/science/journal/01688510

http://www.elsevier.com/locate/healthpol

mailto:[email protected]

mailto:[email protected]


ING Model

lth Polic

stay.We used linear regression for waiting times and length

of stay and logistic regression for whether the waiting time

ARTICLEHEAP-3330; No. of Pages 6

2 A. Scott, J. Yong / Hea

by the Victorian Department of Health (DH, formerly theDepartment of Human Services) under a state-wide work-force redesign initiative called the Better Skills, Best Care(BSBC) strategy. The initiative seeks to encourage hospi-tals to explore new and amended workforce roles, withthe aims of providing better outcomes for patients, pro-moting greater work satisfaction among hospital staff, andcontributing to increased efficiency in service delivery.

The pilot program at Hospital A trialed the new roles ofClinical Assistant and Operations Assistant. The roles weredesigned to reduce the proportion of non-clinical tasks,such as patient scheduling, transcription, billing service,changing linen skips and cleaning trolleys, etc., performedby nurses and doctors. These roles were developed viaa consultation process involving hospital staff and man-agement; the process included identifying tasks requiredto deliver care, analysing skills required to perform thosetasks, reassigning or delegating tasks to new or existingroles, creating revised process maps of patient journeys,and specifying job descriptions and required competencylevels for the new roles.

This evaluation differs from previous workforceredesign evaluation in that it not only compares out-comes before and after the pilot, but also with a group ofcomparable hospitals. The evaluation presents a usefultool for assessing the workforce changes in relation tothe 4-h waiting time target introduced by the Victoriangovernment in 2005/2006. To the best of our knowledge,this study is the first application of the difference-in-differences method in the context of evaluating new orredesigned health workforce roles. Previous studies ofhealth workforce skill mix typically employ methods suchas semi-structured interviews, focus group interviews, andcase study design; see the reviews by Sibbald et al. [9] andHorrocks et al. [10]. Studies evaluating the CWP programin the UK is no exception, semi-structured interviews,case studies and before-after comparisons have been thefavored approaches, e.g., Hyde et al. [7] and Adams et al.[11].

The aim of this paper is to examine the effects of thenew roles introduced under the pilot, which commenced inJuly 2008 and ended in June 2009. Although the pilot lastedapproximately 1 year, some elements of the pilot have sub-sequently been incorporated into the routine operationof the ED. Thus post-pilot data do not provide any usefulbenchmarks for comparisons.

This paper uses routine administrative data to examinethe effects of the new workforce roles on ED waiting timesand subsequent inpatient stay. This evaluation focuses oncomparing outcomes before the pilot with those during thepilot, and with similar outcomes in ‘control’ or comparisonhospitals (i.e., those not participating in the pilot program).

2. Methods

2.1. Study design


This paper uses a difference-in-differences designwhich is the same as a controlled before-and-after design[12,13]. The approach requires two groups of hospitals(Hospital A and comparison hospitals) and at least two time

PRESSy xxx (2014) xxx–xxx

periods (pre-pilot and pilot periods). For the purpose of thisanalysis, the period before the pilot was designated as 1 July2007 to 30 June 2008, and the period during the pilot wasdesignated as 1 July 2008 to 30 June 2009. The analysis esti-mates the difference between the change in outcome overthe two time periods in Hospital A, and the change in theoutcome over the two time periods for the comparison hos-pitals. Taking the difference in these changes removes anypermanent differences between Hospital A and comparisonhospitals, as well as any biases from a simple before–aftercomparison as long as these biases were present in bothperiods.

The Victorian DH identified three regional hospitals thatbest matched Hospital A’s characteristics to serve as a con-trol group. As ‘control’ for the pilot at Hospital A, the samedata for the same time periods were obtained for the com-parison hospitals.1 It is worth noting that the comparisonhospitals were retrospectively chosen during the evalua-tion phase. They were chosen based on having similar sizeand location characteristics as Hospital A. As such they arelikely to differ from Hospital A in other ways (and these maynot be observed in the data), and which may influence thedifference in outcomes between Hospital A and the com-parison hospitals. For example, the comparison hospitalsexperienced similar increases in staffing as Hospital A dur-ing the pilot, but as explained below this increase occurredacross different staff types. Ideally, more careful matchingof hospitals and/or patients should have been undertakenat the design stage of the pilot study.

2.2. Outcome measures

In Victoria, ED presentations are assessed and catego-rized into triage groups according to the Australian TriageScale (ATS). The outcome measures were chosen so thatthe impact of the pilot program could be assessed in rela-tion to the triage categories. For the purpose of this study,three outcome measures were used to evaluate the effectof the pilot program: (i) waiting time (in minutes) at EDbefore treatment, defined as the time between arrival whenthe patient was first registered or triaged and the time thepatient was seen by a doctor or nurse. (ii) Waiting time intarget, a binary outcome since the waiting time is eitherin target or not in target. For each triage category, there isan associated waiting time target (see below). A patient ina given triage category is deemed ‘in target’ if his or herwaiting time is below the set target for the triage category,otherwise the patient’s waiting time is considered ‘not intarget.’ (iii) Length of stay, defined only for ED presenta-tions that led to in-patient admissions, is the number ofdays of in-patient stay. This is used as an outcome measurebecause the timeliness and appropriateness of treatmentwhile in ED may affect the subsequent length of in-patient


1 Ethics approval for using patient admission records from these hos-pitals was obtained by the authors through the Human Research EthicsCommittee of the University of Melbourne.


ARTICLE IN PRESSG ModelHEAP-3330; No. of Pages 6

A. Scott, J. Yong / Health Policy xxx (2014) xxx–xxx 3

Table 1Workforce composition – Hospital A and comparison hospitals.

Comparison hospitals (average) Comparison hospitals (average)

Before pilot FTE During pilot FTE Change in FTE Before pilot FTE During pilot FTE Change in FTE

Administration and clerical staff 3.2 2.1 −1.1 2.7 2.1 −0.6Medical specialist 3.4 3.4 0.0 0.9 1.8 0.9Medical officer 1.0 1.0 0.0 0.5 0.9 0.4Registered nurse 16.1 16.2 0.1 10.3 13.9 3.6Nurse manager/Asst. manager 5.6 5.6 0.0 5.0 5.0 0.1Admin support PCA 0.0 3.2 3.2 0.0 0.0 0.0Clinical support PCA 0.0 3.3 3.3 0.0 0.0 0.0Nurse practitioner 1.6 1.6 0.0 0.0 0.0 0.0

Total 30.9 36.4 5.5 19.3 23.8 4.5

N s the th

wtcTd

asotEc

•

•

•

•

•

fe(op

ote: Comparison hospital FTEs were computed as simple averages acros

as in target or not.2 Regression analysis enables us to con-rol for the different mix of patients between Hospital A andomparison hospitals over the pre-pilot and pilot periods.he covariates used were listed in Table 3 and include gen-er, age and nature of injury.

An important consideration in conducting the statisticalnalysis is that the outcome measures will vary sub-tantially across triage categories. Thus comparisons ofutcome changes should be restricted to within the sameriage category. According to the Victorian DH guideline,D presentations are classified into the following five triageategories, each with a mandated target waiting time.

Triage category 1 (target: immediate)Patients in this group are critically ill and require

immediate attention.Triage category 2 (target: 10 min)

Patients in this group will probably be suffering a crit-ical illness or very severe pain.Triage category 3 (target: 30 min)

Patients in this group include those suffering fromsevere illnesses, people with head injuries but who areconscious, and people with major bleeding from cuts,major fractures, persistent vomiting or dehydration.Triage category 4 (target: 60 min)

Patients in this group usually have less severesymptoms or injuries, although the condition may bepotentially serious.Triage category 5 (target: 120 min)

Patients in this group usually have minor illnesses orsymptoms that may have been present for more than aweek, like rashes or minor aches and pains. The groupincludes people with stable chronic conditions who areexperiencing minor symptoms.

The data on Hospital A and three comparison hospitalsor the two time periods (pre-pilot and pilot periods) were


xtracted from the Victorian Emergency Minimum DatasetVEMD). In total 48,285 and 94,695 patient records werebtained from respectively Hospital A and the three com-arison hospitals. For patients whose ED presentations led

2 The technical details are outlined in Appendix A.

ree comparison hospitals.

to in-patient admissions, their linked admission recordswere also extracted from the Victorian Admitted EpisodesDataset (VAED). Respective totals of 9368 and 12,557 in-patient admission records were extracted from Hospital Aand the three comparison hospitals.

3. Results

Hospital A is located in a small regional city in Victo-ria; it serves a population of about 100,000. In recent yearsthe hospital has been providing care for 17,000 in-patientadmissions and 24,000 ED attendances per year. It employsclose to 1200 people who fill about 840 full-time equiva-lent (FTE) positions, around half of which were nursing staff(about 420 FTEs).

The evaluation focuses on the piloted roles introducedin the hospital’s ED workforce, which had respectively 30.9and 36.4 FTEs before and during the pilot. Table 1 presentsthe workforce composition of Hospital A’s ED before andduring the pilot; for comparison the ED workforce compo-sition of the three comparison hospitals is also included.The FTE figures of comparison hospitals were computed asaverages across the three hospitals; hospital specific FTEscan be found in Appendix B (Table A1).

The introduction of the new roles added 6.5 FTEs to Hos-pital A’s payroll and resulted in the retirement of one FTE,thus a net increase of 5.5 FTEs was recorded and increasedhospital staffing cost by about A$300,000. In the payroll thenew roles were classified as administration support andclinical support Patient Care Attendants (PCA). There wereno other changes in staffing, apart from a slight reductionin administrative staff.

Like Hospital A, the comparison hospitals are locatedin regional areas in Victoria, have similar patient mixand offer similar service type. They each service apopulation of 40,000–45,000 people, and have been pro-viding care to about 11,000–14,000 in-patient admissionsand 16,000–19,000 ED presentations per year. In termsof employment, these hospitals each employed about500–650 FTEs. Thus on the whole each of these hospitals is


slightly smaller in size than Hospital A. It is known thatthese hospitals did not participate in any workforce tri-als and were not involved in any pilot programs that maycontaminate the result of this evaluation.



4 A. Scott, J. Yong / Health Policy xxx (2014) xxx–xxx

Table 2Number and percent of patients – Hospital A and comparison hospitals.

Hosp A before % Hosp A after % Comp hosp before % Comp hosp after %

Triage Cat. 1 42 0.2 47 0.2 112 0.2 108 0.2Triage Cat. 2 1321 5.5 1309 5.4 1346 2.9 1756 3.7Triage Cat. 3 7361 30.5 7333 30.4 9231 19.7 10,857 22.7Triage Cat. 4 11,721 48.6 11,953 49.5 23,274 49.7 23,923 50.0

14.5

100

Triage Cat. 5 3687 15.3 3511

All Cat. 24,132 100 24,153

As shown in Table 1, the three comparison hospitalson average experienced an increase of 4.5 FTEs duringthe period; it is not dissimilar to the increase of 5.5 FTEsrecorded by Hospital A. However, the increase in FTEsoccurred across different skill types within the ED work-force. For example, under registered nursing, Hospital Arecorded an increase of 0.1 FTE whereas the comparisonhospitals registered an average increase of 3.6 FTEs.

On the demand for ED services, Table 2 provides adetailed breakdown of the number of observations undereach triage category for Hospital A and comparison hos-pitals. The distribution of ED presentations across the fivetriage categories is very similar for the two time periods,and also for Hospital A and hospitals in the comparisongroup. Not surprisingly, there were only a small number ofpresentations under Triage Category 1, i.e., presentationsthat required immediate attention.

Table 3 compares the characteristics of patients in Hos-pital A and comparison hospitals. The gender and age ofpatients are similar over the two time periods as well asacross hospitals. The incidences of the types of injury variedsomewhat across the two time periods for both Hospital Aand comparison hospitals.3 Variables listed in Table 3 werealso included in the regression models as covariates to con-trol for differences in patient profile over time and acrosshospitals.

The results from the regression models are presentedusing a series of horizontal bar charts in Fig. 1; the lengthof the bar represents the 95% confidence interval of theestimate, the point estimate is indicated with a solid dotin the middle of the bar. The reference line of no effect ismarked with a red vertical line.

Fig. 1(a) shows the estimates of the piloted workforceroles’ effects on waiting time. The results suggest that thepiloted roles lowered waiting time for all Triage Categories.The effects were the strongest for Triage Categories 4 and5, which experienced a decline in waiting time by about10–11 min on average. The smallest improvement was reg-istered by Triage Category 1. However, since patients inthis category were given immediate attention before andduring the trial and in all hospitals in the sample, it is nosurprise that any improvement here would be small.

Corresponding to Fig. 1(a) and (b) shows strong evi-dence of improvements in the proportions of waiting time


in target. Note that the effect cannot be estimated for TriageCategory 1 since all ED presentations were in target for allhospitals in the sample before and during the trial periods

3 Further descriptive statistics comparing outcome measures of Hospi-tal A and comparison hospitals are included in Appendix B.

12,904 27.5 11,184 23.4

46,867 100 47,828 100

for this category. For Triage Categories 2–5, the evidencesuggested an improvement in in-target proportions thatranged from 5 to 12% points; interestingly the improve-ment appeared to be declining steadily as one moves fromTriage Categories 2–5. The sliding pattern in both Fig. 1(a)and (b) provides strong evidence that the effects are notconstant across triage categories.

A rather mixed picture emerges for the next outcomemeasure, length of stay of patients whose ED presentationsled to in-patient admission. Fig. 1(c) shows that an increasein length of stay was recorded for Triage Categories 1 and2, while a decrease was found for Triage Categories 3–5.However, only the estimates for Categories 3 and 5 are sta-tistically significant from zero; for these two categories, thedecline in length of stay was respectively 0.5 day and 1.1day on average.

4. Discussion

This paper evaluates the effect of introducing two newED workforce roles under the pilot program conducted at aregional hospital in Victoria. Using routine administrativedata, the evaluation makes use of three outcome measures:waiting times before treatment, waiting time in target,and in-patient length of stay. A difference-in-differencesregression analysis was separately carried out for each out-come measure and each triage category. The decision toconduct the analysis by triage categories is driven by thebelief that the effect of the trial was unlikely to be constantacross triage categories. This belief has been verified in theestimates obtained.

The results suggest that the new workforce rolesappeared to have statistically significant and positiveeffects on the ED operation at the hospital. In particular,the trial appeared to have lowered waiting time and raisedthe proportion of ED presentations with waiting time in tar-get. These positive effects appeared to be the strongest forTriage Categories 3–5 which registered a decline in waitingtime of about 5–10 min on average, and an increase in theproportion of waiting time on target by about 6–8% pointson average. The evidence on in-patient length of stay washowever mixed, in some instances an increase in length ofstay was detected while in others a decline was found.

The results however should be interpreted with theassumption of the difference-in-differences approach andthe shortcoming of the study design in mind. The


difference-in-difference method requires that the trialednew roles be the only event that affects the change inthe difference of an outcome over time and across hos-pitals. The validity of the results would be undermined



A. Scott, J. Yong / Health Policy xxx (2014) xxx–xxx 5

Table 3Covariates, means and standard deviations.

Hosp A before Hosp A after Comp hosp before Comp hosp after

Mean s.d. Mean s.d. Mean s.d. Mean s.d.

Male 0.53 0.50 0.53 0.50 0.53 0.50 0.52 0.50Age 35.2 25.0 35.6 25.0 38.7 26.0 38.8 25.9

Nature of main injurySuperficial 0.019 0.137 0.014 0.119 0.030 0.171 0.027 0.161Open wound 0.059 0.235 0.060 0.237 0.063 0.243 0.058 0.234Fracture 0.039 0.194 0.044 0.204 0.036 0.186 0.035 0.183Sprain or strain 0.059 0.235 0.057 0.233 0.034 0.182 0.035 0.184Injury to muscle or tendon 0.009 0.092 0.011 0.106 0.016 0.127 0.018 0.132Foreign body injury 0.015 0.122 0.018 0.133 0.017 0.127 0.016 0.124

18

ietnLtoaathe

cdbt

Others 0.006 0.076 0.0

f, for example, there were events (e.g., outbreak of dis-ases) that led to an increase in ED presentations duringhe trial period and only in areas served by Hospital A butot in the comparison hospitals during the same period.ikewise, the results should be interpreted with caution ifhere were actions (e.g., introducing new work practicesr new technologies) taken by Hospital A, independent ofnd unrelated to the pilot, during the trial period and thesections were not undertaken by comparison hospitals. Inhis case, the change in the difference over time and acrossospitals (for a given outcome) would then include theffect of such unobserved events and/or actions.

A related shortcoming of the study design is that thehoice of comparison hospitals was made retrospectively


uring the evaluation phase of the pilot. Their selection wasased on their comparable size and similar type of loca-ion, as such they are likely to be different to Hospital A

Fig. 1. Estimates of intervention effects by

0.134 0.004 0.061 0.023 0.149

in ways that may not be observed, and which may influ-ence the difference in outcomes between Hospital A andthe comparison hospitals. The comparison hospitals alsoexperienced increases in staffing of a similar magnitudeduring the pilot, but this occurred across different typesof staff than Hospital A, including more medical special-ists and nurses which would have increased costs morethan in Hospital A. Whereas Hospital A employed addi-tional staff to explicitly release the time of nursing andjunior medical staff, the comparison hospitals employedmore nurses and medical staff. The employment of moremedical and nursing staff in the comparison hospitals mayreflect a slightly higher increase in ED presentations anda higher proportion of patients in the more urgent tirage


categories in the comparison hospitals (Table 2). Ideally,more careful matching of hospitals and/or patients shouldhave been undertaken at the beginning of the pilot. This

difference-in-differences analysis.


ING Model

lth Polic

[

[

ARTICLEHEAP-3330; No. of Pages 6

6 A. Scott, J. Yong / Hea

emphasises the importance of planning for the evaluationbefore pilot studies are implemented.

To ensure that the changes in ED workforce are as sim-ilar as possible across all hospitals, the ideal design wouldinvolve one of the two possibilities. The design could ensureeither the workforce budget of comparison hospitals isincreased by the same amount as Hospital A, or an iden-tical increase in FTEs occur in similar workforce skill typesand grades for all comparison hospitals.4 Either of thesepossibilities, however, requires that the comparison hospi-tals be determined before the start of the pilot. Even then,however, there would be practical difficulties for the studyto dictate changes either in workforce size or budget ofcomparison hospitals whose participation in the pilot wasvoluntary. Often practical considerations such as patientcare and hospital workflow would likely take precedenceover the study design.

For the present study, the estimated effect likely rep-resents the combined effects of changes in workforce sizeand the introduction of new workforce roles. However, theshortcoming probably presents a minor failing rather thana major flaw given that the comparison hospitals did onaverage record a similar although not identical net increasein workforce size as Hospital A.

Although the results provide positive evidence that newworkforce roles can be an effective way of improving effi-ciency in the delivery of emergency care, it should beemphasized that the processes involved in designing anddeploying the roles can also be crucial in their eventualsuccess. This study unfortunately cannot differentiate goodprocesses from bad ones, except to note that considerablecare was given to the design and implementation of thenew roles at Hospital A. The piloted roles were met withwide approval by the hospital’s staff and management suchthat substantial elements of the roles were retained afterthe pilot.

This study also did not examine the effects on patientoutcomes or costs. The question of costs raises the issueabout the sustainability of such pilot programs and theireventual state-wide implementation. As Health WorkforceAustralia funds an increasingly large number of workforceinnovation pilot programs, it is important to systematicallyand rigorously evaluate the impact of these pilots on costs


and health outcomes of patients. The use of administrativedata that can potentially be linked to data on quality of careand outcomes is a fruitful way to do this, in addition to localevaluations of such changes.

4 We are grateful to an anonymous reviewer for pointing this out.

[

[

PRESSy xxx (2014) xxx–xxx

Acknowledgements

This article has been developed from the consultancywork undertaken by the authors for the Victorian Depart-ment of Health. The authors would like to acknowledge thehelp of Kathleen Philip and Holly Hildebrand of the Depart-ment. The authors remain solely responsible for any errorsand omissions in this article. The view expressed in thisarticle does not necessarily reflect the view of the VictorianDepartment of Health.

Appendix A. Supplementary data

Supplementary data associated with this article can befound, in the online version, at http://dx.doi.org/10.1016/j.healthpol.2014.11.018.

References

[1] Cameron PA. Hospital overcrowding: a threat to patient safety? Med-ical Journal Australia 2006;184(5):203–4.

[2] Braitberg G. Emergency department overcrowding: dying to get in?Medical Journal Australia 2007;187(11/12):624–5.

[3] Lammy D. Reforming emergency care for patients. EmergencyMedicine Journal 2003;20:112.

[4] Dickson EW, Anguelov Z, Vetterick D, Eller A, Singh S. Use of lean inthe emergency department: a case series of 4 hospitals. Annals ofEmergency Medicine 2009;54(4):504–10.

[5] O’Connell TJ, Bassham JE, Bishop RO, Clarke CW, Hullick CJ, King DL,et al. Clinical process redesign for unplanned arrivals in hospitals.Medical Journal Australia 2008;188(6):S18–22.

[6] McGrath KM, Bennett DM, Ben-Tovim DI, Boyages SC, Lyons NJ,O’Connell TJ. Implementing and sustaining transformational changein health care: lessons learnt about clinical process redesign. MedicalJournal Australia 2008;188(6):S32–7.

[7] Hyde P, McBride A, Young R, Walshe K. Role redesign: new ways ofworking in the NHS. Personnel Review 2005;34(6):697–712.

[8] Callander EJ, Schofield DJ. Emergency department workforce mod-els: what the literature can tell us. Emergency Medicine Australia2011;23:84–94.

[9] Sibbald B, Shen J, McBride A. Changing the skill-mix of thehealth care workforce. Journal of Health Services Research & Policy2004;9(Suppl. 1):28–38.

10] Horrocks S, Anderson E, Salisbury C. Systematic review ofwhether nurse practitioners working in primary care can provideequivalent care to doctors. British Medical Journal 2002;324(8):19–23.

11] Adams A, Wright K, Cooke M. Evaluation of the NHS ChangingWorkforce Programme’s Emergency Care Practitioners Pilot Studyin Warwickshire. Short Report, Division of Health in the Community,Medical School, University of Warwick; 2005.


12] Rubin DB. Estimating causal effects of treatments in randomizedand nonrandomized studies. Journal of Educational Psychology1974;66:688–701.

13] Rubin DB. Bayesian inference for causal effects: the role of random-ization. Annals of Statistics 1978;6:34–58.


http://dx.doi.org/10.1016/j.healthpol.2014.11.018

http://dx.doi.org/10.1016/j.healthpol.2014.11.018

http://refhub.elsevier.com/S0168-8510(14)00333-9/sbref0005












































































































































































































































































































Documents

Do new workforce roles reduce waiting times in ED? A difference-in-difference evaluation using hospital administrative data