ARTICLE IN PRESS

0020-7489/$ - se

doi:10.1016/j.ijn

$Some of the

Quality in Heal�Correspond

Uppsala, Swede

E-mail addr

International Journal of Nursing Studies 44 (2007) 723–735

www.elsevier.com/locate/ijnurstu

Evaluation of an educational ‘‘toolbox’’ for improvingnursing staff competence and psychosocial work environmentin elderly care: Results of a prospective, non-randomized

controlled intervention$

J.E. Arnetza,b,�, H. Hassona,c

aDepartment of Public Health and Caring Sciences, Section for Social Medicine, Uppsala University, Uppsala, SwedenbDepartment of Family Medicine, Division of Occupational and Environmental Medicine, Wayne State University,

Detroit, Michigan, USAcThe Vardal Institute, Lund University, Lund, Sweden

Received 4 November 2005; received in revised form 20 January 2006; accepted 28 January 2006

Abstract

Background: Lack of professional development opportunities among nursing staff is a major concern in elderly care

and has been associated with work dissatisfaction and staff turnover. There is a lack of prospective, controlled studies

evaluating the effects of educational interventions on nursing competence and work satisfaction.

Objectives: The aim of this study was to evaluate the possible effects of an educational ‘‘toolbox’’ intervention on

nursing staff ratings of their competence, psychosocial work environment and overall work satisfaction.

Design: The study was a prospective, non-randomized, controlled intervention.

Participants and settings: Nursing staff in two municipal elderly care organizations in western Sweden.

Methods: In an initial questionnaire survey, nursing staff in the intervention municipality described several areas in

which they felt a need for competence development. Measurement instruments and educational materials for improving

staff knowledge and work practices were then collated by researchers and managers in a ‘‘toolbox.’’ Nursing staff

ratings of their competence and work were measured pre and post-intervention by questionnaire. Staff ratings in the

intervention municipality were compared to staff ratings in the reference municipality, where no toolbox was

introduced.

Results: Nursing staff ratings of their competence and psychosocial work environment, including overall work

satisfaction, improved significantly over time in the intervention municipality, compared to the reference group. Both

competence and work environment ratings were largely unchanged among reference municipality staff. Multivariate

analysis revealed a significant interaction effect between municipalities over time for nursing staff ratings of

participation, leadership, performance feedback and skills’ development. Staff ratings for these four scales improved

significantly in the intervention municipality as compared to the reference municipality.

e front matter r 2006 Elsevier Ltd. All rights reserved.

urstu.2006.01.012

results presented in this manuscript were presented as an oral presentation at the 22nd International Conference for

th Care in October, 2005 in Vancouver, Canada.

ing author. Department of Public Health and Caring Sciences, Uppsala University, Uppsala Science Park, SE-751 85

n. Tel.: +46 18 611 35 92; fax: +46 18 51 16 57.

ess: judy.arnetz@pubcare.uu.se (J.E. Arnetz).

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735724

Conclusions: Compared to a reference municipality, nursing staff ratings of their competence and the psychosocial

work environment improved in the municipality where the toolbox was introduced.

r 2006 Elsevier Ltd. All rights reserved.

Keywords: Elderly care; Competence development; Intervention; Nursing; Staff training; Work satisfaction

What is already known about the topic?

�

Lack of professional development opportunities

among nursing staff is a major concern in elderly

care and has been associated with work dissatisfac-

tion, staff turnover and quality of care problems.

�

Competence development has been pointed out as

one important area for improving work satisfaction

among elderly care nursing staff.

�

There is a lack of prospective, controlled studies

evaluating the effects of educational interventions on

nursing competence and work satisfaction.

What this paper adds

�

This is among the first prospective, controlled studies

investigating the effects of an educational interven-

tion on both the self-rated competence and psycho-

social work environment of elderly care nursing staff.

�

Elderly care workplaces provided with a collation of

practical instruments had significantly improved

ratings of nursing staff competence and the psycho-

social work environment, compared to reference

worksites.

1. Introduction

Lack of professional development opportunities

among nursing staff is a major concern in elderly care

and has been associated with work dissatisfaction

(Brodaty et al., 2003) and staff turnover (D’Eramo

et al., 2001; Sung et al., 2005). Licensed practical nurses

and nurses’ aides make up the largest groups of staff

providing direct care for the elderly, and both these

groups are less skilled and educated than other health

care providers (D’Eramo et al., 2001; Pennington et al.,

2003). While the proportion of elderly people in many

countries is increasing (Thorslund and Parker, 1995;

Oberski et al., 1999), elderly care recipients today also

have more complex medical needs than previously,

placing new demands on nursing staff competence (Sung

et al., 2005; Proctor et al., 1999; Robertson and

Cummings, 1996). In particular, increases in psychiatric

morbidity, physical disability and the prevalence of

dementia imply a need for more specialized training for

nursing care staff (Proctor et al., 1999; Pennington et al.,

2003). There is a need for continual competence and

skills’ development among nursing staff in elderly care in

order to increase work satisfaction, decrease turnover,

and uphold the quality of care. For example, an inverse

association has been shown between turnover of nursing

staff and certain quality indicators, including risk for

infection and hospitalization (Zimmerman et al., 2002)

and increased prevalence of contractures and pressure

ulcers (Castle and Engberg, 2005).

A positive association between nursing staff partici-

pation in continuing education activities and ratings of

work satisfaction has been reported (Robertson et al.,

1999; Sung et al., 2005). In a study of nurse self-assessed

competence, Tzeng (2004) suggested that opportunities

for learning would contribute to organizational commit-

ment among nursing staff. However, previous educa-

tional interventions among elderly care nursing staff

have rarely been evaluated with scientific rigor. In their

review of 48 studies on continuing education in long-

term care, Aylward et al. (2003) found that most studies

were limited both in methodological design and evalua-

tion. Only 26 of the 48 studies used randomized

controlled trial or quasi-experimental designs. Of these

26, 14 studies reported only immediate post-intervention

results, with no follow-up data. Validity was often

questionable due to small sample sizes, lack of

randomization, lack of control groups and low response

rates. In 35 of the 48 studies, educational interventions

were implemented without any organizational or work-

place support to facilitate or maintain the new knowl-

edge or activity. The authors conclude that there is a

need for more rigorous research on educational inter-

ventions in long-term care, and that the role of

organizational factors—both facilitators and hin-

drances—must be considered (Aylward et al., 2003).

Few studies have considered the possible impact of

educational interventions on staff work environment

and satisfaction. Using qualitative analysis, Haggstrom

et al. (2005) studied work satisfaction in nursing staff

following an educational intervention. Despite a quasi-

experimental design, this particular study reported only

before and after results in the intervention worksite,

with no comparison to the control site.

There is a lack of prospective, controlled studies

evaluating the effects of educational interventions on

nursing competence and work satisfaction. Beginning in

2001, elderly care in one Swedish municipality became

the focus of such an intervention. Elderly care services in

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735 725

the municipality were marked by serious employment

problems, including high rates of sickness absenteeism

and high turnover among nursing staff. The educational

intervention was designed so that it could be utilized at

the individual workplace according to the need for

competence and skills’ development among the nursing

staff at that particular site. This intervention offered a

selection of tools, rather than a standardized program,

in an effort to maximize workplace interest and

participation.

The present study describes the development and

evaluation of the educational intervention. The inter-

vention, called the ‘‘toolbox,’’ was actually a collation of

practical instruments and educational materials that

could be used by nursing staff at their workplaces. The

aim of the study was to evaluate the effects of the

toolbox intervention on nursing staff ratings of their

competence, psychosocial work environment and overall

work satisfaction. Compared to a reference municipal-

ity, where no toolbox was introduced, the hypothesis

was that staff in the intervention municipality would

give higher ratings to their competence, psychosocial

work environment and overall work satisfaction follow-

ing introduction and implementation of the toolbox.

2. Methods

2.1. Settings and study participants

Two municipalities in western Sweden were involved

in the toolbox project, one as an intervention site and

the other as a reference. The workplace toolbox,

described below, was introduced only in the intervention

municipality. The reference municipality had no knowl-

edge of, or exposure to, the toolbox. Both municipalities

provide healthcare and general services for their elderly

citizens, with approximately 50% receiving services in

their homes and the others in residential nursing care

facilities. The intervention municipality has approxi-

mately 11,000 inhabitants, of which 4% ðn ¼ 490Þ are

elderly care recipients. The municipality employs ap-

proximately 350 nursing staff, of which 75% work in

nursing home facilities and 25% work in home care. The

reference municipality has 49,500 inhabitants, with 4%

ðn ¼ 1800Þ receiving elderly care. Its elderly care has a

staff of approximately 950, with 65% working in

nursing home facilities and 35% employed in home

care. These two municipalities were recruited for the

project because both had received orders from the

Swedish Work Environment Authority to evaluate the

working situation of their elderly care staff. In both

cases, the orders were prompted by internal reports of

high levels of work strain and dissatisfaction among

nursing staff. In response to the Authority’s demand, the

intervention municipality approached the research team

for help in carrying out a study of the psychosocial work

environment in their elderly care services. A question-

naire study of staff competence and psychosocial work

environment was conducted in the intervention munici-

pality in September 2001. The toolbox intervention was

based on the results of this baseline questionnaire study.

The reference municipality was recruited for this study

specifically as a control site in September 2003, six

months after the toolbox was introduced in the

intervention municipality.

2.2. Baseline questionnaire study

In September of 2001, baseline work environment

questionnaires were distributed in the intervention

municipality to all nursing staff ðn ¼ 355Þ. Nursing staff

was defined as registered nurses, licensed practical

nurses and nurses’ aides. In Sweden, registered nurses

complete a three-year university education, which may

be complemented by additional courses towards a

specialist education, such as elderly care nursing.

Licensed practitioners have a three-year upper second-

ary school education with a focus on nursing. This

education replaced an earlier upper secondary educa-

tional program for nurses’ aides. Those working as

nurses’ aides in Sweden have either received their

training in the previous upper secondary school

program, or completed a 40-week course given by the

Swedish employment office.

A shortened version of the Quality-Work-Compe-

tence (QWC) questionnaire, which has been used

extensively and validated among health care staff in

Sweden (Arnetz, 1997; Arnetz, 1999), was used. The

QWC questionnaire measures the psychosocial work

environment by means of 11 scales: work-related

exhaustion (3 questions), work stress (4 questions),

participation (6 questions), efficiency (4 questions),

leadership (5 questions), work climate (3 questions),

mental energy (5 questions), goal clarity (4 questions),

employeeship (4 questions), performance feedback (3

questions) and skills’ development (4 questions). Each

scale uses a Likert-type response scale, with response

alternatives such as ‘‘disagree strongly, disagree some-

what, agree somewhat, agree completely;’’ and ‘‘often,

sometimes, seldom, never.’’ An overview of the scales

with their component items and Cronbach’s alpha

values for internal homogeneity is presented in Table 1.

Only 6 of the 11 QWC scales were used in the baseline

questionnaire study in the intervention municipality

(Table 1). Scores for all scales were calculated for each

respondent by totalling the scores on the component

items and converting that sum to a percentage of the

maximum possible score. Thus, scores for all scales are

reported as mean percentage scores. For all scales except

for Work stress and Work-related exhaustion, higher

values indicate more positive ratings from staff. Mean

ARTICLE IN PRESS

Table 1

Staff psychosocial work environment questionnaire: scales (Cronbach’s alpha) and component items (number of items)

Scale (Cronbach’s alpha) Component items (n)

Work-related exhaustiona (.86) Feelings of emptiness after work, feelings of exhaustion after work, tired when thinking about

work (3)

Work stressa (.74) Time for planning work tasks in advance, sufficient time to execute tasks, time to reflect upon/

consider how tasks should be carried out, time to consider how work processes could be

improved at one’s workplace (4)

Participationa (.74) Freedom to decide how work should be carried out, freedom to decide what tasks should be

carried out, sufficient authority in relationship to responsibilities, opportunity to influence

workplace decisions, opportunity to comment on the information received from immediate

supervisor, actual influence over your work situation in relationship to how you’d like it to be (6)

Efficiencya (.73) Planning work tasks, employees strive toward the same goals, resources used optimally at work,

the decision-making process works well (4)

Leadershipa (.87) My immediate supervisor: communicates clearly, is consistent in his actions, has described how

to achieve departmental goals, provides me with opportunities to develop my professional skills,

is open for changes in workplace organization and work routines (5)

Work climatea (.88) Positive atmosphere at work, cohesion among co-workers, supportive atmosphere among co-

workers (3)

Mental energy (.85) Feelings during the last month of: restlessness, irritability, worry, moodiness/depression,

difficulty in concentrating (5)

Goal clarity (.90) Workplace goals: are well-defined, are realistic, can be influenced, can be measured (4)

Employeeship (.75) One is open for changes and development of working routines, takes responsibility for

competence/professional development, initiates changes and development at work, takes

responsibility for getting the latest information at work (4)

Performance feedback (.73) Clear work directives from immediate supervisor, feedback from supervisor when a job has been

done well, feedback from supervisor when a job has not been done well (3)

Skills’ development (.77) One’s knowledge is utilized at work, opportunities for professional development, immediate

supervisor provides employee with opportunities for competence development, current job tasks

contribute to one’s professional development (4)

Work satisfaction Overall rating of work satisfaction on a modified visual analogue scale from 1 (not at all) to 10

(very): ‘‘How satisfied are you overall with your work situation?’’

aOnly these six scales were used in the 2001 baseline measurement in the intervention municipality

J.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735726

values for Work stress are considered optimal when they

lie within the range of 33–38%, while means for work-

related exhaustion should not exceed 30% (Anderzen

and Arnetz, 2005). Staff were also asked to rate their

satisfaction with their work on a visual analogue scale

from 1, not at all satisfied, to 10, very satisfied (Table 1).

Mean values for satisfaction ratings were also converted

to percentages and treated as a continuous variable

(scale).

2.3. Competence questions

Competence in this study was measured in two ways:

by staff ratings of their knowledge in specific areas, and

by the scale skills’ development. Additional items in the

baseline questionnaire asked nursing staff to rate their

knowledge in eighteen specific competence areas. Re-

sponse alternatives were ‘‘Sufficient knowledge,’’ ‘‘In-

sufficient knowledge,’’ or ‘‘Don’t know.’’ These

competence questions were the result of a series of focus

group discussions held with staff representatives in the

intervention municipality in 2000, prior to the baseline

measurement. The index skills’ development includes

four items regarding opportunities for professional

development at work (Table 1). For this index higher

values indicate more positive staff ratings of skills’

development.

2.4. Development of the toolbox intervention

A total of 270 (76%) staff responded to the

questionnaire, reporting a need to improve their

competence in several areas. These included information

about specific medications, knowledge about dementia,

oral hygiene for the elderly and strategies for handling

aggressive patients. As a result, a work group comprised

of researchers and elderly care management representa-

tives collated a ‘‘toolbox’’ of practical instruments for

use at elderly care workplaces. Some of the toolbox

instruments were meant to improve staff knowledge in

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735 727

specific areas while others were designed to help staff in

various aspects of their daily work. A total of 16

instruments were included in the toolbox, organized into

four categories: (1) instruments for improving the

working situation for nursing staff; (2) educational

materials; (3) instruments for meeting elderly residents’

social and physical needs; and (4) questionnaires used in

the intervention municipality in 2001 to measure staff

Table 2

Overview of instruments included in the workplace toolbox

Description

Instruments for improving nursing staff work situation

3-3 work scheduling Innovative work schedu

Violent incident form (VIF) Checklist for recording

Video films Filmed sequences of sp

tasks for ergonomic or

assessment

Work load assessment Computer program for

recipients’ needs and pe

requirements

Educational materials for nursing staff

Eating and nutrition in elderly care Information and recom

regarding food, nutritio

routines

Infection protection manual Work practices for prev

treatment of gastrointe

Study circle materials on Dementia General information on

related conditions, refe

recommended reading

Study material on Swedish health care

and social welfare laws

Legal texts with explana

Instruments for meeting residents’ social and physical needs

Resident activity list Checklist for recording

preferences for social a

activities

Monitors Computer program for

medicine consumption,

compatibility and drug

problems

Oral Health Instrument Guide for inspection, d

and evaluation of oral

function

Pressure ulcer evaluation card Protocol for prevention

pressure ulcers

Resident assessment instrument (RAI) Assessment of functiona

needs

Questionnaire instruments

Quality-work-competence (QWC) Measures staff-rated ps

environment

Pyramid quality of care questionnaire Measures quality of ca

viewpoint of the care r

Relative questionnaire Measures quality of eld

viewpoint of family rela

work environment, resident-perceived quality of care,

and relative-perceived quality, respectively. These were

accompanied by guidelines for interpreting question-

naire results. Brief descriptions of the toolbox instru-

ments are compiled in Table 2. Whenever possible and

applicable, validated research tools, which comprise

approximately half of the total number of instruments,

were included.

Reference

ling system http://www.tretre.se/korteng.shtml

violent events Arnetz (1998), Arnetz and Arnetz (2000)

ecific nursing

nursing care

info@halsobolaget.se

measuring care

rsonnel resource

Local contact person

mendations

n and mealtime

Local contact person

ention and

stinal infections

Head of nursing in the municipality

dementia and

rence list,

Local contact person

tory information Actual legal documents, National Board

of Health and Welfare, Sweden

www.socialstyrelsen.se

residents’

nd leisure

Kielhofner and Burke (1980)

analysis of

drug

-related health

Pharmaceutical Division, National

Board of Health and Welfare, Sweden

johan.fastbom@sos.se.

ocumentation

health status and

Andersson et al. (2002)

and treatment of Ek et al. (1989)

l ability and care Morris et al. (1990), Sgadari et al. (1997).

ychosocial work Arnetz (1997)

re from the

ecipient

Arnetz and Arnetz (1996), Ygge and

Arnetz (2001)

erly care from the

tives

Verho and Arnetz (2003)

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735728

2.5. Implementation of the toolbox intervention

One-page descriptions of each of the toolbox inter-

vention instruments were collated in a binder. Each

description stated the instrument’s purpose, use, cost

(when applicable) and the name of a contact person for

further information. Whenever possible, the actual

instrument or educational material was included in the

binder; otherwise, the description was accompanied by

appropriate references. In February of 2003, a total of 37

binders were distributed—one binder per workplace—in

conjunction with a seminar organized by the research

team. Each workplace had the freedom to decide to what

extent they would work with the instruments provided.

No general guidelines regarding work with the instru-

ments were given by the researchers or the elderly care

managers in the municipality. Reference municipality

workplaces did not receive the toolbox binder or any

information about the instruments.

2.6. Study design

This study was a prospective, non-randomized con-

trolled intervention with questionnaire measurements in

the intervention municipality at baseline (September

2001) and six and 18 months, respectively, after the

introduction of the toolbox intervention (February

2003). Two measurements were made in the reference

municipality, in September 2003 (reference baseline) and

September 2004 (12-month follow-up). See Fig. 1 for a

timeline of the project.

Staff ratings of their psychosocial work environment

were measured in each municipality with the QWC

questionnaire, described above. At all measurement

periods, questionnaires were distributed at work to all

elderly care nursing staff currently employed. Work-

place managers at each respective workplace distributed

questionnaires directly to each employee. Questionnaires

were mailed home to staff that had been on maternity or

sick leave for less than three months. All staff that had

been away from the workplace for longer than three

months, due to maternity or sick leave, were excluded

from the study. Questionnaire responses were anon-

ymous and no reminders were sent. Each individual was

given an addressed, postage-paid envelope in which to

return the questionnaire to the research unit.

At each measurement period, questionnaire results for

each municipality, respectively, were summarized in

Sept. 2001 Feb. 2003

Baseline Toolboxmeasurement (I) introduced (I)

Fig. 1. Timeline for toolbox intervention proje

written reports and presented orally at a single meeting

of elderly care managers from all workplaces.

2.7. Statistical analysis

Chi square statistics were used to compare staff

ratings of the knowledge questions in 2003 and 2004

both within and between municipalities. These analyses

were based only on positive (‘‘sufficient’’) and negative

(‘‘insufficient’’) responses to the questions, i.e. all ‘‘don’t

know’’ responses, which most often accounted for less

than 10% of responses, were excluded.

Staff questionnaires in 2003 and 2004 in the interven-

tion municipality also included a question concerning

staff experience in working with the intervention

instruments. Response alternatives to this question were

‘‘Yes’’, ‘‘No,’’ and ‘‘Don’t know.’’ Chi square statistics

were used to study changes in responses to these

questions between 2003 and 2004.

Staff ratings of the 12 psychosocial work environment

scales were examined for normality using the Kolmo-

gorov–Smirnov test. Since not all scales were normally

distributed, both non-parametric (Mann–Whitney) and

parametric tests were performed, yielding identical

results. Thus, changes over time in nursing staff ratings

of the psychosocial work environment were measured in

the intervention municipality by one-way analysis of

variance (ANOVA) with post hoc tests using Bonferro-

ni, and in the reference municipality by independent

sample T-tests. Multivariate ANOVA was used to study

possible interaction effects between municipalities re-

garding changes in the psychosocial work environment

over time. In these analyses, all 12 scales were the

dependent variables, with municipality and measure-

ment year as fixed factors. These multivariate analyses

were based on data from the 2003 and 2004 measure-

ments only.

2.8. Power calculation

Since the study concerned an educational interven-

tion, calculation for necessary group size was based on

the scale for measuring Skills’ development (Table 1). At

least 180 respondents per measurement would be needed

in order to detect an increase between measurements in

ratings of this scale from a means of 58% to a means of

67% with a power of 100% (Alpha ¼ 0.05, two-tailed).

Sept. 2003 Sept. 2004

6-month 18-monthmeasurement (I) measurement (I)

Baseline 12-monthmeasurement (R) measurement (R)

ct: (I) ¼ intervention; (R) ¼ reference.

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735 729

The power to detect a difference of 9% in mean values

for Skills’ development between municipalities with 180

respondents in each group was also 100% (Alpha ¼

0.05, two-tailed). These calculations were based on an

expected increase in mean values for Skills’ development

in the intervention municipality from the 50th percentile

ðmeans ¼ 58Þ to the 70th percentile ðmeans ¼ 67Þ.

Statistical significance for all analyses was set at po:05(two-tailed). The SPSS statistical package for personal

computers, version 12.0 was used for all analyses.

Ethical approval for this multi-centre project was

granted by the research ethics committee of the main

research institution as well as its regional affiliate at the site

of the participating municipalities (dossier number 00-206).

3. Results

Questionnaire response rates and characteristics of

nursing staff respondents to the questionnaires in both

Table 3

Response rates and characteristics of nursing staff respondents to psyc

Municipality Sept. 2001

I n (%) Ra n (%)

n (response rate) 270 (76)

Age (yrs.)

p 39 96 (36)

40–49 82 (31)

X 50 86 (33)

Professionb

Reg. Nurse 17 (6)

Practical nurse 149 (55)

Nurse’s aide 104 (39)

Type of workplacec

Home care 64 (24)

Nursing home 198 (76)

Years in professiond

0–5 46 (17)

6–20 143 (53)

X 21 80 (30)

Years at current workplacee —f

o1

1–5

6–20

X 21

I ¼ intervention; R ¼ referenceaNo questionnaire study carried out.bpo:05 2003 I vs. R.cpo:01 2004 I vs. R.dpo:001 2004 I vs. R.epo:001 2003 and po:001 2004 I vs. R.fitem not included in questionnaire.

municipalities are summarized in Table 3. Since 99% of

all nursing staff in both municipalities were women, the

questionnaires did not include any question regarding

gender. The response rate in the intervention munici-

pality was above 75% in 2001 and above 85% in both

2003 and 2004. Response rates in the reference

municipality were just under 70% in both 2003 and

2004. Since recruitment of new staff was limited in both

municipalities during the time of the study, it can be

assumed that respondents were more or less the same

throughout the data collection period.

3.1. Self-rated knowledge

Nursing staff ratings of their knowledge in specific

areas were compared in the intervention municipality in

2003 and 2004, 6 and 18 months, respectively, post-

intervention. Statistically significant improvements were

found in ratings of 10 out of 18 competence questions,

i.e., a greater percentage of intervention group staff

hosocial work environment questionnaires: 2001, 2003 and 2004

Sept. 2003 Sept. 2004

I n (%) R n (%) I n (%) R n (%)

273 (87) 606 (67) 213 (81) 647 (69)

84 (31) 207 (35) 56 (27) 207 (32)

87 (32) 158 (26) 64 (30) 188 (29)

100 (37) 237 (39) 90 (43) 246 (39)

6 (2) 37 (6) 18 (8) 62 (10)

156 (57) 325 (54) 121 (57) 357 (55)

111 (41) 244 (40) 74 (35) 228 (35)

75 (28) 223 (37) 20 (10) 255 (40)

192 (72) 373 (63) 191 (90) 383 (60)

61 (23) 142 (24) 22 (10) 127 (20)

117 (43) 299 (50) 101 (48) 327 (51)

93 (34) 159 (26) 89 (42) 187 (29)

16 (6) 73 (12) 14 (7) 75 (12)

91 (33) 233 (39) 53 (25) 240 (37)

125 (46) 246 (41) 106 (50) 282 (44)

41 (15) 50 (8) 38 (18) 44 (7)

ARTICLE IN PRESS

Table 4

Nursing staff self-rated knowledge, 2003 vs. 2004 I ¼ intervention; R ¼ reference

Competence area Municipality 2003 2004 Percentage point

change 2003–

2004 within

municipality

p-value

within

municipality

2003 2004

Adequate

knowledge

n (%)

Adequate

knowledge

n (%)

p-value

between

municipalities

p-value

between

municipalities

Administrationa I 168 (75) 160 (85) +10 0.019 ns 0.023

R 428 (80) 444 (77) �3 ns I4R

Computer skills I 51 (21) 39 (21) 0 ns ns ns

R 111 (22) 148 (27) +5 0.045

Dementia I 83 (34) 92 (48) +14 0.004 0.002 ns

R 255 (46) 273 (45) �1 ns IoR

General patient care I 253 (95) 203 (97) +2 ns ns ns

R 566 (96) 599 (96) 0 ns

Ethical issues I 187 (76) 153 (78) +2 ns ns ns

R 460 (82) 509 (83) +1 ns

Patient transfers I 211 (80) 175 (85) +5 ns ns ns

R 467 (81) 505 (82) +1 ns

Threats and

violence

I 53 (22) 57 (31) +9 ns 0.003 ns

R 174 (33) 189 (34) +1 ns IoR

Laws regarding

healthcare and

social welfare

I 56 (25) 67 (37) +12 0.009 0.000 0.030

R 227 (42) 273 (47) +5 ns IoR IoR

Leadership I 53 (25) 63 (36) +11 0.014 ns ns

R 144 (31) 181 (33) +2 ns

Medication I 99 (39) 110 (55) +16 0.001 0.000

R 315 (56) 357 (59) +3 ns IoR ns

Psychiatric illness I 416 (16) 46 (23) +7 ns 0.001 ns

R 152 (27) 163 (27) 0 ns IoR

Contractures/

limited range of

motion

I 124 (57) 121 (63) +6 ns ns ns

R 304 (59) 342 (60) +1 ns

Infection protection I 108 (46) 126 (63) +17 0.000 0.020 0.036

R 305 (55) 323 (54) �1 ns IoR I4R

Pressure ulcers I 150 (60) 155 (75) +15 0.001 ns 0.001

R 365 (65) 382 (63) �2 ns I4R

Carrying out

delegated tasks

I 266 (88) 185 (93) +5 ns ns ns

R 511 (90) 541 (90) 0 ns

Palliative care I 154 (62) 160 (78) +16 0.000 0.030 0.026

R 396 (69) 427 (70) +1 ns IoR I4R

Oral care/hygiene

for elderly residents

I 200 (78) 181 (86) +8 0.039 ns ns

R 464 (81) 494 (80) �1 ns

Diet and nutrition

for elderly residents

I 158 (64) 162 (79) +15 0.001 0.001 ns

R 427 (75) 457 (75) 0 ns IoR

ns ¼ non-significant.aCalling in substitutes, scheduling staff, ordering supplies, etc.

J.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735730

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735 731

rated their knowledge as adequate in 2004 as compared

to 2003. In the reference municipality, staff self-rated

knowledge improved significantly over time in only one

area, computer skills (Table 4). In 2003, nursing staff

ratings differed significantly between municipalities on 8

of the 18 competence questions. In all 8 areas, a

significantly smaller proportion of intervention nursing

staff rated their knowledge as adequate, compared to the

reference group (Table 4). In 2004, nursing staff ratings

differed significantly between municipalities in only 5

competence areas. In four of the five areas—adminis-

tration, infection protection, pressure ulcers, and pallia-

tive care—a significantly greater proportion of

intervention group staff rated their knowledge as

adequate, compared to reference group staff. In the

fifth area, regarding healthcare and social welfare laws,

the proportion of staff rating their knowledge as

adequate remained larger among reference group staff.

3.2. Psychosocial work environment

Nursing staff ratings of their psychosocial work

environment in both municipalities are compared in

Table 5. There were significant improvements over time

in the intervention municipality for 9 out of 12 work

environment scales. Staff ratings of both work-related

exhaustion and work stress decreased significantly,

although mean values for exhaustion still exceeded the

recommended level of 30% at all measurement periods.

Mean values for participation, efficiency, leadership,

work climate, performance feedback, skills’ development

and work satisfaction all showed significant increases. Six

of the psychosocial work environment scales were

measured on three occasions in the intervention munici-

pality: in 2001 (baseline) and at 6-months (2003) and 18-

months (2004) post-intervention. Post-hoc tests per-

formed on the analyses of variance for these six scales

revealed steady and significant improvements over time in

all six areas (Table 5). There were no significant changes

over time in nursing staff ratings of their psychosocial

work environment in the reference municipality.

The multivariate analysis revealed a significant inter-

action effect between municipalities over time

(2003–2004) for nursing staff ratings of participation,

leadership, performance feedback and skills’ develop-

ment. This indicates that there was a statistically

significant difference between the municipalities in

nursing staff ratings of these four scales over time. Staff

ratings for all four scales improved significantly in the

intervention municipality as compared to the reference

municipality.

3.3. Work with toolbox instruments

In 2003, 42% ðn ¼ 106Þ of questionnaire respondents

in the intervention municipality reported that their

workplace had worked with toolbox instruments; 32%

ðn ¼ 82Þ had not, and 26% ðn ¼ 66Þ responded ‘‘don’t

know.’’ In 2004, 53% ðn ¼ 111Þ reported having worked

with instruments, 27% ðn ¼ 55Þ had not, and 20% ðn ¼

42Þ didn’t know. This increase in utilization of the

toolbox was statistically significant ðpo:05Þ.

4. Discussion

In the intervention municipality, nursing staff ratings

of both their competence and their psychosocial work

environment, including work satisfaction, improved

significantly over time. In the reference municipality,

staff rated only one of the 18 competence questions,

regarding computer skills, significantly higher in the

follow-up measurement in 2004, while work environ-

ment ratings were unchanged. The significant differences

in competence ratings between the municipalities in 2003

revealed significantly more positive ratings among

reference group nursing staff in 8 of 18 areas. By 2004,

the significant differences were reduced to 5 areas, and in

4 of these, it was a greater percentage of intervention

group staff who rated their knowledge as adequate.

Moreover, the significant differences between munici-

palities in 2003 in the areas of dementia, threats and

violence, medication, psychiatric illness and diet and

nutrition were no longer significant in 2004. This

indicates a significant improvement in intervention

group ratings of their competence. Significant interac-

tion effects between the municipalities over time for four

psychosocial work environment scales offer further

evidence of different developments within each organi-

zation. These findings suggest that the introduction of

the educational toolbox was related to the improve-

ments reported in the intervention municipality.

The competence areas that showed improvements in

intervention staff ratings (Table 4) could all be

associated with toolbox instruments, with the exception

of palliative care. It is possible that intervention staff

attended lectures or courses on this subject, unbe-

knownst to the research team. A personal communica-

tion with an administrator in the reference municipality

revealed that reference group staff did participate in

computer skills’ training in 2004, which might explain

that significant improvement in our findings.

There were consistently positive trends over time in

intervention staff ratings of all aspects of their psycho-

social work environment. Mean values for work-related

exhaustion and work stress both decreased while all

other areas increased. With the exception of mental

energy, goal clarity and employeeship, all changes over

time were statistically significant. The multivariate

analysis showed statistically significant differences be-

tween municipalities over time for four areas—partici-

pation, leadership, performance feedback and skills’

ARTICLE IN PRESS

Table 5

Nursing staff ratings of the psychosocial work environment, mean percentage scores. SD ¼ standard deviation, Intervention (I) vs.

Reference (R) over time

Scale Municipality 2001 Means

(SD)

2003 Means

(SD)

2004 Means

(SD)

Significance

level within

municipality

over time

Interaction

effect

municipality �

yearg

Work-related exhaustion I 52.6 (24.1) 46.6 (24.6) 44.1 (24.3) po:01a ns

R — 43.6 (24.8) 44.2 (25.2) ns

Work stress I 40.0 (19.7) 32.7 (20.7) 29.6 (20.2) po:001b ns

R — 30.5 (21.0) 30.5 (20.7) ns

Participation I 62.7 (20.0) 62.9 (20.9) 70.4 (19.0) po:001c po:05R — 69.4 (20.4) 68.9 (20.0) ns

Efficiency I 65.0 (20.3) 69.0 (17.6) 71.8 (18.6) po:01d ns

R — 69.8 (18.1) 68.9 (18.4) ns

Leadership I 46.5 (26.3) 49.8 (27.2) 65.7 (21.9) po:001e po:001R — 66.6 (22.5) 64.0 (23.8) ns

Work climate I 72.2 (22.9) 76.5 (21.5) 79.0 (19.8) po:01f ns

R — 71.7 (21.7) 70.8 (22.5) ns

Mental energy I — 75.4 (22.9) 77.5 (22.8) ns ns

R — 75.9 (21.4) 75.9 (21.6) ns

Goal clarity I — 44.8 (29.9) 50.0 (30.7) ns ns

R — 56.9 (27.0) 57.6 (24.6) ns

Employeeship I — 76.3 (15.9) 77.9 (15.5) ns ns

R — 79.0 (15.8) 79.9 (15.8) ns

Performance feedback I — 40.3 (30.2) 59.0 (30.0) po:001 po:001R — 56.3 (27.0) 56.9 (27.2) ns

Skills’ development I — 55.1 (20.6) 65.0 (19.6) po:001 po:01R — 64.7 (21.6) 63.8 (21.4) ns

Work satisfaction I — 64.6 (24.3) 69.2 (23.2) po:05 ns

R — 68.0 (23.2) 67.9 (24.2) ns

ns ¼ non-significant.apo:001 2001–2003; po:01 2001–2004.bpo:001 2001–2003 and 2001–2004.cpo:001 2001–2003 and 2001–2004.dpo:01 2001–2003; po:001 2001–2004.epo:001 2001–2003 and 2001–2004.fns 2001–2003; po:01 2001–2004.gWilk’s Lambda ¼ .972 (p ¼ :001).

J.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735732

development. Three of these scales—participation,

performance feedback and skills’ development—relate

to work tasks, professional skills, and the way in which

work tasks are carried out. Skills’ development, in

particular, concerns opportunities for professional

development. It is possible that the improvements in

these ratings can be attributed to the toolbox interven-

tion. Eighteen months after the introduction of the

toolbox, intervention staff were clearly more positive

about all aspects of their work, as well as their self-rated

competence in several areas. Overall work satisfaction

also increased significantly among intervention staff. No

corresponding improvements were found in the refer-

ence municipality.

To our knowledge, this is among the first prospective,

controlled studies investigating the effects of an educa-

tional intervention on self-rated competence, the psy-

chosocial work environment and work satisfaction of

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735 733

elderly care nursing staff. Numerous reports in the

literature have pointed to the need for substantial

improvements in these three areas in order to safeguard

the quality of elderly care (Proctor et al., 1999;

Robertson et al., 1999; Robertson and Cummings,

1996). The quantitative nature of the study enabled us

to survey several hundred nursing staff. One of the

strengths of the current study is that the intervention

was carried out among all nursing staff in one

municipality’s elderly care services and that staff ratings

of their work situation were measured before and six

and 18 months, respectively, after the introduction of

the toolbox intervention. Few intervention studies have

studied post-intervention changes in self-rated compe-

tence as well as changes in work satisfaction among

elderly care nursing staff. In their review article of

educational interventions in long-term care, Aylward et

al. (2003) showed that most interventions were evaluated

on the basis of either staff behaviour, staff attitudes or

resident outcomes. Staff were not asked to rate their

psychosocial work environment or work satisfaction

before and/or after the training programs in any of the

48 studies in their report. In a qualitative study based on

ten narratives from nursing staff, Haggstrom et al.

(2005) found a trend toward greater work satisfaction

following a two-year intervention including education,

support and clinical supervision in a single nursing

home. Although the current study was quantitative, our

findings were similar to those of Haggstrom et al. (2005).

However, Haggstrom et al. (2005) evaluated their

intervention using before and after measurements only.

This was a ‘‘bottom up’’ intervention, with the needs

for competence development defined by the staff in their

responses to the baseline questionnaire in 2001. While

the toolbox concept had been developed by the research

team during project conception, it was the nursing staff

themselves who helped to determine what kinds of tools

would be included. Haggstrom et al. (2005) also

developed an intervention based on staff requests that

aimed to improve the working situation for staff. One of

the main reasons for our design was to motivate staff to

use the instruments that they felt best suited the needs of

their respective workplaces. On the other hand, work-

places were not required to use any of the instruments

and had complete freedom to decide. There was a

statistically significant increase in the percentage of

intervention nursing staff who had worked with work-

box tools in 2004, compared to the year before.

However, the current study only compared worksites

exposed to the toolbox (the intervention municipality)

with non-exposed worksites (reference municipality). It

would be interesting in future studies to investigate

whether improvements in work environment and/or

competence ratings were greater at workplaces that used

the toolbox to a greater extent. The focus of the current

study was to compare the group exposed to the toolbox

with the group that was not exposed. A closer

examination of the actual amount of toolbox use among

the exposed would demand a different study design and

will be the subject of future research.

In their study of organizational factors associated

with the effectiveness of continuing education in long-

term care, Stolee et al. (2005) emphasized the impor-

tance of workplace environments that supported staff

efforts to try innovative approaches to improving their

knowledge and skills. A crucial factor was management

commitment to empower staff to change practice. The

toolbox notebooks were made available to all nursing

staff at all intervention worksites and gave both staff

and workplace managers something concrete on which

to base their own competence development initiatives.

Pennington et al. (2003) found that certified nursing

assistants preferred ‘‘onsite education to improve skills

for providing better resident care’’ over formal educa-

tion to a higher professional level (p. 583). The results

presented here indicated a steady improvement over

time in intervention staff ratings of their knowledge and

work situation. Further research is needed in order to

better understand factors that contribute to the facilita-

tion and sustainability of these positive changes.

4.1. Limitations

One of the main difficulties in evaluating interventions

of this kind is that it is impossible to rule out the possible

effects of other changes occurring during the three-year

study period. Secondly, while the toolbox was intro-

duced only in the intervention municipality, both

municipalities were, in fact, intervention sites in that

questionnaire studies were carried out in both. Elderly

care management in both municipalities also received

reports on questionnaire results. The intervention

municipality received feedback from three questionnaire

measurements, compared to only two in the reference

municipality. This may have added to any ‘‘Hawthorne

effect’’ that was potentially affecting intervention nur-

sing staff, i.e., the very fact that the questionnaire studies

were being carried out may have been enough to

improve staff work satisfaction (Roethlisberger and

Dickson, 1939). Any such effect would actually decrease

the likelihood of detecting differences between groups.

However, no corresponding improvements in work

satisfaction ratings were found among reference munici-

pality staff, despite the fact that two questionnaire

studies were carried out there. Intervention staff were

aware of the toolbox and this may have biased their

questionnaire responses. However, as mentioned earlier,

the percentage of staff working with the toolbox

increased significantly between 2003 and 2004, which

would help to explain the improved competence ratings.

As illustrated in Fig. 1, this study did not follow a

strict design since baseline measurements for the two

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735734

municipalities did not coincide. The 2003 measurement

was a baseline measurement for the reference organisa-

tion, while the intervention had already been underway

for 6 months in the intervention municipality. However,

it was deemed advantageous to include the reference

municipality in the study for two reasons. First, both

municipalities were dealing with similar work environ-

ment issues, and both were subjected to mandatory

work environment assessments. Secondly, inclusion of

the reference municipality enabled comparison of out-

come data between municipalities. The length of the

project, from 2001 to 2004, enabled us to follow the

reference municipality prospectively, with two question-

naire measurements. This design was deemed preferable

to a single cite before-after design because it offered an

external organisation as a means of comparison. A

closer comparison of work environment scales between

the two organisations in 2003 revealed better staff

ratings in the reference organisation for all scales except

for Work climate. By 2004, when the intervention had

been in effect for 18 months, intervention staff work

environment ratings were equal to or better than

reference staff ratings in all areas except goal clarity.

Thus, the results of this study revealed improvements

between 2003 and 2004 for all outcome variables only in

the intervention group. Results also indicated significant

improvements in the intervention group between 2001

and 2003. The lack of a control group for that point in

time is a limiting factor. However, there were indications

that improvements were already underway.

The two municipalities differed in sample size and

there were significant differences between questionnaire

respondents on some background factors. Carrying out

the toolbox intervention at half of the workplaces within

a single municipality would have been an alternative

method, but would have increased chances of response

bias due to ‘‘leakage’’ of information about the

intervention between nursing staff and managers at

different workplaces. While basic differences between

the municipalities could potentially affect study results,

the fact remains that the toolbox intervention was

implemented at each individual work site. The interven-

tion municipality had fewer worksites than the larger

reference municipality, but this fact would not explain

the significant improvements seen over time in interven-

tion staff ratings of their competence and work

situation.

5. Conclusions

Compared to a reference municipality, nursing staff

ratings of their competence and psychosocial work

environment improved in the municipality where the

toolbox was introduced. An educational intervention of

this type is designed to be used at the discretion of the

individual workplace. It is suggested that a standardized

educational toolbox, adapted to local needs, might be an

important and sustainable intervention to improve job

skills, work conditions and job satisfaction in elderly

care.

Acknowledgements

This study was financially supported by The Swedish

Agency for Innovation Systems VINNOVA, The

Swedish Foundation for Health Care Sciences and

Allergy Research (The Vardal Foundation), Swedish

Skandia Life (publ) Group, and the two municipalities

involved in the project. The authors would like to thank

Professor Bengt Arnetz for valuable comments on this

manuscript.

References

Andersson, P., Hallberg, I.R., Renvert, S., 2002. Inter-rater

reliability of an oral assessment guide for elderly patients

residing in a rehabilitation ward. Special Care in Dentistry

22 (5), 181–186.

Anderzen, I., Arnetz, B.B., 2005. The impact of a prospective

survey-based workplace intervention program on employee

health, biologic stress markers, and organizational produc-

tivity. Journal of Occupational and Environmental Medi-

cine 47 (7), 671–682.

Arnetz, B.B., 1997. Physicians’ view of their work environment

and organisation. Psychotherapy and Psychosomatics 66,

155–162.

Arnetz, J.E., 1998. The Violent Incident Form (VIF): a

practical instrument for the registration of violent events

in the health care workplace. Work & Stress 12 (2), 17–28.

Arnetz, B., 1999. Staff perception of the impact of health care

transformation on quality of care. International Journal for

Quality in Health Care 11, 345–351.

Arnetz, J.E., Arnetz, B.B., 1996. The development and

application of a patient satisfaction measurement system

for hospital-wide quality improvement. International Jour-

nal for Quality in Health Care 8, 555–565.

Arnetz, J.E., Arnetz, B.B., 2000. Implementation and evalua-

tion of a practical intervention programme for dealing with

violence towards health care workers. Journal of Advanced

Nursing 31 (3), 668–680.

Aylward, S., Stolee, P., Keat, N., Johncox, V., 2003. Effective-

ness of continuing education in long-term care: a literature

review. The Gerontologist 43 (2), 259–271.

Brodaty, H., Draper, B., Low, L., 2003. Nursing home staff

attitudes towards residents with dementia: strain and

satisfaction with work. Journal of Advanced Nursing 44

(6), 583–590.

Castle, N.G., Engberg, J., 2005. Staff turnover and quality of

care in nursing homes. Medical Care 43 (6), 616–626.

D’Eramo, A.L., Papp, K.K., Rose, J.H., 2001. A program on

complementary therapies for long-term care nursing assis-

tants. Geriatric Nursing 22 (4), 201–207.

ARTICLE IN PRESSJ.E. Arnetz, H. Hasson / International Journal of Nursing Studies 44 (2007) 723–735 735

Ek, A.C., Unosson, M., Bjurulf, P., 1989. The modified Norton

scale and the nutritional state. Scandinavian Journal of

Caring Sciences 3 (4), 183–187.

Haggstrom, E., Skovdahl, K., Flackman, B., Kihlgren, A.L.,

Kihlgren, M., 2005. Work satisfaction and dissatisfaction—

caregivers’ experiences after a two-year intervention in a

newly opened nursing home. Journal of Clinical Nursing 14,

9–19.

Kielhofner, G., Burke, J., 1980. A model of Human Occupa-

tion, part one. American Journal of Occupational Therapy

34, 578–586.

Morris, J.N., Hawes, C., Fries, B.E., Phillips, C.D., Mor, V.,

Katz, S., Murphy, K., Drugovich, M.L., Friedlob, A.S.,

1990. Designing the national resident assessment instrument

for nursing homes. Gerontologist 30 (3), 293–307.

Oberski, I.M., Carter, D.E., Gray, M., Ross, J., 1999. The

community gerontological nurse: themes from a needs

analysis. Journal of Advanced Nursing 29 (2), 454–462.

Pennington, K., Scott, J., Magilvy, K., 2003. The role of

certified nursing assistants in nursing homes. Journal of

Nursing Administration 33 (11), 578–584.

Proctor, R., Burns, A., Stratton Powell, H., Tarrier, N.,

Faragher, B., Richardson, G., Davies, L., South, B., 1999.

Behavioural management in nursing and residential homes:

a randomised controlled trial. Lancet 354, 26–29.

Robertson, J.F., Cummings, C.C., 1996. Attracting nurses to

long-term care. Journal of Gerontological Nursing 22 (9),

24–32.

Robertson, E.M., Higgins, L., Rozmus, C., Robinson, J.P.,

1999. Association between continuing education and job

satisfaction of nurses employed in long-term care facilities.

Journal of Continuing Education in Nursing 30 (3),

108–113.

Roethlisberger, F.J., Dickson, W.J., 1939. Management and the

Worker. Harvard University Press, Cambridge, MA.

Sgadari, A., Morris, J., Fries, B., Ljunggren, G., Jonsson, P.,

DuPaquier, J., Schroll, M., 1997. Efforts to establish the

reliability of the Resident Assessment Instrument. Age and

Ageing 26 (Suppl. 2), 27–30.

Stolee, P., Esbaugh, J., Aylward, S., Cathers, T., Harvey, D.P.,

Hillier, L.M., Keat, N., Feightner, J.W., 2005. Factors

associated with the effectiveness of continuing education in

long-term care. The Gerontologist 45 (3), 399–405.

Sung, H., Chang, S., Tsai, C., 2005. Working in long-term care

settings for older people with dementia: nurses’ aides.

Journal of Clinical Nursing 14, 587–593.

Thorslund, M., Parker, M.G., 1995. Strategies for an ageing

population: expanding the priorities discussion. Ageing and

Society 15, 199–217.

Tzeng, H.-M., 2004. Nurses’ self-assessment of their nursing

competencies, job demands and job performance in the

Taiwan hospital system. International Journal of Nursing

Studies 41, 487–496.

Verho, H., Arnetz, J.E., 2003. Validation and application of an

instrument for measuring patient relatives’ perspective of

quality of geriatric care. International Journal for Quality in

Health Care 15 (3), 197–206.

Ygge, B.-M., Arnetz, J.E., 2001. Quality of paediatric care:

application and validation of an instrument for measuring

parent satisfaction with hospital care. International Journal

for Quality in Health Care 13, 33–43.

Zimmerman, S., Gruber-Baldini, A.L., Hebel, J.R., Sloane,

P.D., Magaziner, J., 2002. Nursing home facility risk factors

for infection and hospitalization: importance of registered

nurse turnover, administration, and social factors. Journal

of the American Geriatric Society 50 (12), 1987–1995.