Children and Youth Services Review 34 (2012) 289–295

Contents lists available at SciVerse ScienceDirect

Children and Youth Services Review

j ourna l homepage: www.e lsev ie r .com/ locate /ch i ldyouth

Measuring organizational effectiveness to develop strategies to promote retention inpublic child welfare

Crystal Collins-Camargo a,⁎, Chad D. Ellett b, Cathy Lester c

a University of Louisville, United Statesb CDE Research Associates, Inc., United Statesc Kentucky Department for Community Based Services, United States

⁎ Corresponding author.E-mail address: crystal.collinscamargo@louisville.ed

0190-7409/$ – see front matter © 2011 Elsevier Ltd. Alldoi:10.1016/j.childyouth.2011.10.027

a b s t r a c t

a r t i c l e i n f o

Article history:Received 21 July 2011Received in revised form 25 October 2011Accepted 29 October 2011Available online 6 November 2011

Keywords:Organizational effectivenessEmployee retentionIntent to remain employedChild welfare

Public child welfare agencies are under pressure to improve organizational, practice and client outcomes. Re-lated to all of these outcomes is the retention of staff. Employee intent to remain employed may be used as aproxy for actual retention. In this study public child welfare staff in one Midwestern state were surveyedusing the Survey of Organizational Excellence (Lauderdale, 1999) and the Intent to Remain Employed (Ellett,Ellett, & Rugutt, 2003) scales to assess the extent to which constructs such as perceptions of organizationalculture, communication and other areas of organizational effectiveness were associated with intent to remainemployed. A number of statistically significant relationships were identified which were presented to thepublic agency for use in the development of strategies for organizational improvement. Data were also ana-lyzed regionally and based on urban/suburban/rural status to enable development of targeted approaches.This case study presents an example of how ongoing measurement of organizational effectiveness can beused as a strategy for organizational improvement over time in the child welfare system.

© 2011 Elsevier Ltd. All rights reserved.

1. Introduction

The issue of staff turnover in public child welfare agencies has beenwell documented. The American Public Human Services Associationsurvey of states revealed national turnover rate of 22.1% for child pro-tection workers, with states reporting up to 67% (Cyphers, 2005). Theliterature has documented a number of factors associated with turn-over. To summarize, Zlotnick, DePanfilis, Daining, and Lane (2005a,b)systematic review of 29 recent child welfare studies focused on reten-tion found both personal factors (professional commitment, previouswork experience, education, job satisfaction, efficacy, burnout, emo-tional exhaustion, and role overload/conflict/stress) and organizationalfactors (salary, reasonable workload, supervisory support, coworkersupport, opportunities for advancement, and valuing workers) contrib-uted to retention. Some of these factors are within the control of the or-ganization to adjust without significant resources, while others are not.

More recently focus has shifted to measurement of intent to re-main employed, and the identification of factors associated withthose who remain employed as opposed to those who leave employ-ment in child welfare (Ellett, 2000; Ellett, Collins-Camargo, & Ellett,2006; Ellett et al., 2003; Landsman, 2001). Methodological issues as-sociated with studying actual retention and turnover in child welfareand other human services agencies has been previously noted (Mor

u (C. Collins-Camargo).

rights reserved.

Barak, Nissly, & Levin, 2001; Zlotnick et al., 2005a,b). Some studieshave found that intention to leave is the strongest predictor of actualturnover (Alexander, Lichtenstein, Oh, & Ullman; 1998; Mor Barak etal., 2001). The measurement of such proxy constructs in conjunctionwith factors which are within the locus of control of child welfareagencies makes sense if they seek to retain competent, committedstaff.

Agencies have significant reasons for wanting to promote retention.Staff turnover is expensive. One study calculated a cost of $10,000 pervacancy in 1995 dollars (Graef & Hill, 2000). The impact of turnoveron children and families receiving services has been widely discussed(e.g. U.S. General Accounting Office, 2003). Staff turnover in thishighly stressful yet critical field tends to have a snowball effect.Responding to crises in uncovered caseloads of vacant positions fallsto the staff remaining, multiplying their workload and their frustration.It is certainly in the best interest of agencies to identify those factorsmost powerfully associated with intent to remain employed withtheir particular staff, assess for regional differences, and to use this in-formation to plan strategies based on addressing those factors so asnot to waste limited resources.

Thefield of childwelfare has been under appropriate pressure tomea-sure and improve outcomes for children and families through the Childand Family Services Review process (U.S. General Accounting Office,2004). Many states continue to struggle to improve their performancein these reviews. Although achieving timely permanency for children infoster care is a federally mandated priority, of the 32 states that hadcompleted their Child and Family Services Review in 2007 and 2008 an

290 C. Collins-Camargo et al. / Children and Youth Services Review 34 (2012) 289–295

average of 40% of cases reviewed across states achieved PermanencyOutcome 1: children have permanency and stability in their livingsituations (Administration for Children and Families, n.d.). Emphasis isneeded, in practice and in research related to which strategies on thefrontline and organizational level promote outcome achievement(Barth, 2008).

Public child welfare organizations are faced with constant changedue to social and political influences, and the complicated needs ofthe client population (Cohen & Austin, 1994; Morrison, 1997). Childwelfare agencies are increasingly more focused on improving perfor-mance and accountability (Weigensberg, 2009), yet research into theability of the public child protection system to achieve its outcomeshas been scant (Poertner, McDonald, & Murray, 2000; Waldfogel,2000). Agencies with limited resources must determine where tofocus their efforts within the organization in order to improve the abil-ity and likelihood of frontline staff to provide effective services accord-ing to their selected practicemodel aswell as evaluate the impact of theservices being provided on outcomes.

A number of studies have linked organizational culture and climateto the achievement of client outcomes and organizational effectivenessin a variety of human service settings (Glisson, 2007; Glisson, Dukes, &Green, 2006; Glisson & Hemmelgarn, 1998). Workplace climate hasbeen shown to influence openness to change and innovation(Anderson & West, 1998; Birleson, 1999). There is a need for furtherstudy of how to assess social context in organizations, the dimensionsof social context that are related to the implementation of effective ser-vices, and the identification of organizational interventions that can cre-ate the sort of social contexts that support the implementation ofeffective services (Glisson, 2007).

This article describes a study in which one largely rural state useda standardized measure of organizational effectiveness as an overallassessment of the agency and its functioning from the perspectiveof their staff, and paired it with a measure of intent to remainemployed. Of particular focus was the organizational culture of theagency and the extent to which organizational culture would corre-late with a measure of intent to remain in child welfare. Over two de-cades ago Schein defined organizational culture as “a pattern of basicassumptions – invented, discovered, or developed by a given group asit learns to cope with its problems of external adaptation and internalintegration – that has worked well enough to be considered valid andtherefore, to be taught to new members as the correct way to per-ceive, think, and feel in relation to the problems.” (1985, p. 9) For or-ganizational culture, the unit of analysis is the organization or largedepartments within it as an entity as opposed to the work unit(Burke and Litwin, 1992). However in large agencies, organizationalculture is translated, interpreted, and selectively observed withinthe frontline work unit (Cohen & Austin, 1994; Martin & Siehl,1983). Research has positively linked organizational culture to childwelfare staff retention (Ellett & Millar, 2001); quality of services, con-sumer satisfaction, and client outcomes (Glisson & Hemmelgarn,1998), and self-efficacy in child welfare tasks and case outcomes(Collins-Camargo, 2005).

Recently, Shim (2010) found the differential effect of organizationalculture, specifically in the construct of organizational emphasis on re-wards, and organizational climate, specifically in the level of emotionalexhaustion of staff. Organizational climate has been defined as staff'sshared perception of their work environment (e.g. Verbeke, Volgering,& Hessels, 1998), and Glisson recently categorizes it into stressful, func-tional or engaged climates (2007) which could be interpreted to bemost influencedwithin the frontlinework unit reporting to a supervisorwho in turn helps promote such a tone. In the current study, aspects ofboth culture and climate are most likely at play, however, focus of theanalysis is primarily on those constructs which may be influenced onthe organizational and regional, rather than frontline, levels.

Based on the literature, and the state child welfare organization'sdesire to improve both employee outcomes such as retention and

ultimately outcomes for children and families such as safety, perma-nency and well-being, the study was designed on an assumptionthat organizational life, culture and climate are related to employeeoutcomes (in this case, intent to remain employed) and subsequentlyto outcomes for children and families (based on employee retentionand the provision of quality services). While these outcomes werenot measured in the current study, in the interaction between agencyadministrators and researchers, this assumed relationship was impor-tant to organizational motivation to participate in the study and con-sider strategies in response to findings. The development and testingof organizational interventions to address organizational contributorsto staff turnover in child welfare has been described by Collins-Camargo, Sullivan, Washeck, Adams, and Sundet (2009) as an effec-tive approach to organizational change.

The purpose of this study was to examine the linkages betweenorganizational constructs and intent to remain employed, with partic-ular attention to the utility of survey measurement for use by childwelfare administrators as a tool to promote organizational, and ulti-mately although not measured here, outcome improvement. Specifi-cally, this study tests the hypothesis that there was a statisticallysignificant positive relationship between subscales of the Survey ofOrganizational Excellence (SOE) and the Intent to Remain Employed(IRE) scale (Ellett, 2000). Secondly, it involves a case study regardingthe use of study findings in collaboration with the public child welfareagency to implement employee retention strategies.

Findings from analysis of the data were presented to agency admin-istrative staff for use in strategic planning regarding overall organiza-tional improvement as well as organizational approaches to retentionof valuable frontline staff with emphasis on those constructs demon-strated to be significantly related to IRE that were amenable to theagency's intervention given current contextual factors such as budget-ary realities and bureaucratic structures. In addition these purposeswhich is the focus of this manuscript, data collection was used for thepurpose of psychometric comparison of twomeasures of organizationalfactors as a part of a largermulti-state study of childwelfare supervision(Collins-Camargo, 2007). Despite the openness with which the agencyapproached the study and its findings, missed opportunities for usingsuch organizational measurement to effect and measure true organiza-tional improvement will be discussed.

2. Methodology

2.1. Sample

This study was completed through the collaborative efforts of thepublic child welfare agency and the university researchers. After re-ceiving approval from both the university and public agency institu-tional review boards, the agency's staff child welfare researchdirector communicated the purpose and relevance of the study to or-ganizational priorities to central office and regional managementstaff, in order to enlist their support in obtaining frontline staff partic-ipation. A purposive sample of all staff (N=1485) employed in thefour job classifications which comprise frontline child welfareworkers was used: social service workers I and II, and social serviceclinicians I and II. The classifications represent four professional ca-reer ladder positions for frontline workers, with eligibility based onyears of experience and educational background, up to but not in-cluding frontline supervisors.

2.2. Measures

Theprimarymeasure usedwas the Survey of Organizational Excellence(SOE) (Lauderdale, 2001). The SOE has demonstrated through multiplestudies to yield reliable data (Cronbach's Alpha of .85 or greater). TheSOE has been under continuous development to enhance its validitycharacteristics including convergent validity studies with other

291C. Collins-Camargo et al. / Children and Youth Services Review 34 (2012) 289–295

measures such as the burnout subscale with Dean's Alienation Scale, andMaslach's Burnout Inventory. In addition, SOE content validity has beenaddressed through expert panel reviews, and comparison of data rank-ings of constructs to ratings of organizational constructs by trained ob-servers (Lauderdale, 1999).

Although not designed exclusively for child welfare agencies, thisSOE is particularly well suited to the present study as it was devel-oped to promote a work environment of self reflection and learningin organizations (Lauderdale, 2001). The authors purport that it mea-sures five workplace dimensions (work group, work setting, organi-zational features, communication and personal demands), withinwhich have been identified twenty constructs, including supervisoryeffectiveness, team effectiveness, organizational change-oriented,goal-oriented, and quality (Lauderdale, 1999).

Constructs are measured with eighty-six items using a six-pointLikert scale (Strongly Disagree = 1, Disagree = 2, Feel Neutral = 3,Agree = 4, Strongly Agree = 5, and Don't Know/Not Applicable = 0,which is not scored). Don't Know/NotApplicable responseswere not in-cluded in the analysis. It has been used over time in other states tomea-sure organizational improvement over time (Collins-Camargo et al.,2009). Examples of items in the SOE include “Weproduce a high qualityof work that has a low rate of error;” “Wedevelop services tomatch ourcustomers' needs;” “Information systems are in place and accessible forme to get my job done;” “The work atmosphere encourages open andhonest communication;” “There is a basic trust among employees andsupervisors;” “There is a real feeling of teamwork;” “Weare encouragedto learn from our mistakes;” “The people I work with treat each otherwith respect;” and, “Work groups are actively involved in makingwork processes more effective.” The SOE was used to collect data re-garding workplace dimensions including organizational culture in achild welfare setting.

The standardized web-based survey platform allows a limited num-ber of additional questions to be inserted to meet an organization'sneeds. The Ellett Intent to Remain Employed Scale (IRE) (Ellett et al.,2003) was added to the SOE electronic survey platform. The IRE wasoriginally developed by Ellett (2000) and has been refined into a ten-item measure. In this study a five-point Likert-style scale was usedrather than the four-point forced choice options used by the authors,in order to conform to the standardized response structure of theweb-based SOE platform. A Principal Components Analysis of IRE re-sponses from a statewide sample of 1423 child welfare employees(Ellett et al., 2003) accounted for 43.2% of the total IRE item varianceand yielded a Cronbach's Alpha reliability coefficient of .85. Examplesof the IRE include “I will remain in child welfare even though I mightbe offered a position outside of child welfare with a higher salary;” “Ifeel the personal and professional gratifications ofworking in childwel-fare are greater than those in other professions;” and, “I would leavechild welfare work tomorrow if I was offered a job for the same salarybut with less stress.” The IRE was included as an outcome proxy mea-sure for remaining employed in the field.

2.3. Data collection procedures

The public child welfare agency provided an email address list forall staff in the four frontline child welfare worker positions. An emailinvitation to participate in the study was sent to 1485 individualsusing a recruitment and informed consent script developedcollaboratively. The email included a hotlink to the electronic survey.By choosing to access the survey and submit it, respondents providedinformed consent. The SOE is administered through the Institute forOrganizational Excellence. Survey responses are collected withintheir databank, are subsequently cleaned, and the raw data is providedto the researcher, stripped of any identifiers. In this way, respondentparticipation is enhanced due to increased confidence that theirresponses cannot be traced back to them.

2.4. Data analyses

Data were analyzed using SPSS to describe the sample, identify bi-variate relationships between factors within the SOE and the intent toremain employed. A variety of data analyses were completed in thestudy. These included 1) descriptive statistics to document characteris-tics of the sample and the SOE and IREmeasures; 2) a series of PrincipalComponents Analyses (PCA) of the SOE data; 3) bivariate correlationsbetween factored scales of the SOE and the uni-dimensional IRE; and4) a series of one way ANOVAs to compare SOE and IRE results byurban, suburban and rural work contexts. Data were also analyzed re-gionally so that those areas of the state with the most and least favor-able findings in factors of organizational excellence could be identifiedso that case study procedures would be possible subsequently to iden-tify what may be contributing to staff's positive perceptions.

3. Findings

3.1. Sample description

In response to the electronic invitation to participate, 680 frontlinestaff completed the survey, which was a response rate of 47%. Partic-ipation was relatively widespread, with staff from 111 of the state's120 counties responding. Given that in some counties only a fewworkers participated, analysis was not conducted on this level, andwas disaggregated to the regional level to preserve the anonymityof respondents.

The demographics of the respondent group were generally typicalof the child welfare workforce. Eighty-three percent of the samplewas female. Ethnicity was reported as follows: 88% Anglo-American/White, 9% African-American/Black, and 3% Other. Over three quartersreported their highest level of educational attainment to be a Bachelorsdegree. Thirty-one% selected an age range of 16–29 years, with another33.4% falling in the 30–39 year category.

The SOE also collects data on a number of work-related character-istics. Over 60% reported a salary of $25,001–35,000, with another34% earning up to $45,000. Three quarters of the respondents hadworked for the child welfare agency for three or more years, withthe remainder having two years or less on the job. Slightly over onequarter indicated they had received a promotion within the pasttwo years. A full 20% indicated they did not intend to be workingfor the agency in two years, reinforcing the importance of this kindof study. It should be noted here that this large percentage did nottake into account respondents who might be close to retirement.

3.2. Findings related to organizational effectiveness

The SOE process yields a standardized report generated by theInstitute for Organizational Excellence of the child welfare agency'sratings on the 20 constructs they have designed the instrument tomeasure across five dimensions: Personal (e.g. job satisfaction andempowerment), Information (e.g. availability and internal), Organi-zational Features (e.g. quality and goal oriented); Accommodations(e.g. fair pay and employee development); and Work Group (e.g.supervisor effectiveness and team effectiveness). From this, organi-zational strengths and opportunities for growth are identified. Forexample, this child welfare agency which was the subject of thisstudy scored high in employment development, the agency's re-sponse to external factors that should play a role in how it defines it-self, information flow to and from external sources, perceptions thata level playing field exists for all staff, and the degree to which qualityprinciples are a part of the agency's organizational culture. Importantopportunities for growth were revealed: perceptions of the employeecompensation package, job satisfaction, the extent to which employeesfeel the job demands are realistic given time and resource limitations,perceptions regarding the flow of information within the organization,

292 C. Collins-Camargo et al. / Children and Youth Services Review 34 (2012) 289–295

and supervisory effectiveness. In addition, the agency received informa-tion on how they scored in comparison to benchmark organizations ofsimilar size and mission, which provides key contextual informationwhich may be useful as an organization considers its own responsescompared to others.

3.3. Intent to remain employed and related organizational factors

The need for attention to staff retentionwas confirmed based on theresponse to the IRE subscale. This scale is not zero-based, and the mid-point, or neutral response is a 3. Of a possible maximummean score of5.0, the mean for the sample was 2.52, standard deviation .283. Thissuggests the risk of staff turnover could be even more significant thanthat suggested by the one item in the SOE measuring the respondents'intention to still beworking for the agency in two years.When analyzedbased on the respondents' experience working in the agency, an evenmore alarming finding was revealed: 49% of those with 2 years or lessexperience had an IRE score in the lower third.

Although the SOE provides data on 20 separate constructs accordingto Lauderdale (2001), a Principal Components Analysis of the data wascompleted in order to establish the statistical relationships among SOEitems and empirically derived measurement components with thissample. Ten SOE components (subscales)were identified as follows (re-liability coefficient shown in parentheses): Interpersonal/OrganizationalCulture—26 items (.96), Internal/External Communications/Relation-ships—11 items (.89); Compensation/Benefits—7 items (.82); Organiza-tional Service and Productivity—7 items (.83); Workload—3 items(.83); Professional Development Opportunities—6 items (.80); Satisfac-tion with Job Benefits—5 items (.79); Information Resources—5 items(.76); Work Community—4 items (.67); and Satisfaction withInsurance—2 items (.83). These PCA components rather than the 20scales of the original SOE were used in subsequent correlation andANOVA analyses.

Bivariate correlation analyses were completed to examine relation-ships between factors of the SOE and the IRE. Although all but Satisfac-tion with Insurance were positively associated with IRE at a statisticallysignificant level, the strongest correlation with the IRE measure waswith SOE Factor I (Interpersonal/Organizational Culture) (r=.73;pb .0001). The second strongest correlation with IRE was withInternal/External Communications and Relationships (r=.62).

3.4. Assessment of geographical variance

Many state child welfare agencies understand the significant dif-ferences in both the provision of services and organizational function-ing in rural, suburban and urban areas (Landsman, 2002). Because

Table 1Summary of descriptive statistics for the Kentucky survey for rural, suburban, and urban groEach factor for the complete survey sample (n=690).

SOE Factors Rurala

(n=203)Mean SD

Interpersonal/organizational culture (26)d 80.88 20.76Internal/external communications and relationships (11) 36.41 7.36Compensation and benefits (7) 18.65 5.39Organizational service/productivity (7) 24.90 4.75Professional development opportunities (6) 21.61 4.64Workload (3) 7.86 2.95satisfaction with job benefits (5) 17.83 3.55Information resources (5) 15.99 3.53Work community (4) 14.47 2.99Satisfaction with insurance (2) 2.95 1.24IRE (10) 16.10 3.35

a County population b25,000.b County population 25,000–80,000.c County population>80,000.d Number of items comprising measure.

work context factors (e.g., nature of clients, community support ser-vices, poverty, quality of law enforcement; involvement of the courts)are known to differ in child welfare work settings, it was of interest inthe study to make comparisons between child welfare staff's perspec-tives in rural, suburban, and urban groups. These groups were definedby county populations as follows: Rural, b25,000; Suburban, 25,000–50,000; Urban, >80,000. Table 1 shows descriptive statistics (Meansand Standard Deviations) for each of the factored dimensions of theSOE measure, and the IRE measure. The final column in Table 1shows the Mean as a percentage of the maximum possible score foreach variable measured. This index allows for rough comparisonsamong the scale scores when they have different numbers of items.It should be noted though, that these are not exact estimates sincethe rating scales were not zero-based.

By way of summary, there are several results worthy of note inthis table. First, child welfare staff in working in urban settings hadlower mean scores on all measurement variables than those workingin suburban and rural areas, including scores on the IRE. Second, andwith a few exceptions, mean scores for child welfare professionalsworking in rural contexts were higher (more positive perceptions)than for those working in suburban or urban contexts.

Third, the standard deviation (SD) statistic can be considered anindex of cohesiveness or agreement in judgments within and acrossgroups for any row comparison. The lower the SD value for a givengroup and measurement dimension, the more cohesive respondentswere in their perceptions of the measurement dimension. For thetable total, 5 SD values were lowest (one was the same) for thoseworking in suburban contexts. In comparing the differences in SDvalues across groups for the different measurement dimensions, thelargest, meaningful differences in SD values were for SOE Factor 2(Internal/External Communications and Relationships) (Rural groupSD compared to Suburban group SD), SOE Factor 5 (ProfessionalDevelopment Opportunities) (Rural group compared to Suburbangroup), SOE Factor 7 (Satisfaction with Job Benefits) (Rural groupcompared to Suburban group),SOE Factor 8 (Information Resources)(Rural group compared to Urban group), and SOE Factor 10 (Satisfac-tion with Insurance (Rural Group compared to Urban group)).

A series of one-way ANOVAs was computed on each of the mea-surement dimensions shown in Table 1 to test for the statistical sig-nificance of the differences between the perceptions of the threework context groups (Rural, Suburban, Urban). The results of theseanalyses showed statistically significant (pb .01) differences for allSOE dimensions and the IRE except for SOE Compensation and Bene-fits, Professional Development Opportunities, and Satisfaction withJob Benefits. The differences were in the direction of the magnitudeof measurement means shown in Table 1.

ups, and mean scores expressed as percentages of the maximum score (Mean%Max) for

Suburbanb

(n=242)Mean SD

Urbanc

(n=245)Mean SD

Mean%Max fortotal sample

78.51 19.58 72.71 19.58 59.3436.12 6.82 33.50 7.51 64.1318.65 5.28 17.88 5.03 52.4924.21 5.03 22.61 5.04 68.1121.57 3.90 21.04 4.03 71.307.81 2.78 7.05 2.76 50.33

18.03 2.96 17.58 3.13 71.2416.06 3.74 14.73 4.16 62.2814.16 2.87 13.42 2.97 69.953.11 1.49 3.70 1.73 32.70

16.08 2.91 15.30 3.06 65.88

293C. Collins-Camargo et al. / Children and Youth Services Review 34 (2012) 289–295

As compensation and benefit packages as well as professional de-velopment and training opportunities are usually established on astatewide level in state-administered child welfare systems, these ex-ceptions are to be expected. The differences are in the direction of themagnitude of measurement means shown in Table 1. These findingsmay inform the agency how differential approaches to organizationalimprovement may be appropriate based on geographical region.

A final analysis was completed for the agency establishing the order-ing of ratings for each of the factors by service region. In this state, a levelof middle management administers child welfare service programsacross a multi-county region. The study was able to identify those re-gions which performed highest and lowest in factors of interest. Particu-lar attention was paid to ratings on the Interpersonal/OrganizationalCulture, Intent to Remain Employed and the overall SOE score. In general,urban and more suburban service regions tended to score lower thanthose with more rural communities. When comparing those service re-gions encompassing rural areas, however, there was a great deal of var-iance noted on most factors. Analyzing the data in this way enabledmanagers in those regions to target those areas in which a desire for im-provementwasnoted. In addition, this comparison provided anopportu-nity for the agency to conduct a more qualitative study of differences inpractice and approach in the high performing regions in an attempt toidentify interventions that may improve perceptions in lower perform-ing areas.

4. Discussion and implications for the field

This study's greatest contribution may be in offering an example ofhow collection and analysis of such data could be useful to organiza-tions as they work to identify organizational issues needing attention,design responses and monitor impact—as well as how organizationalcircumstances can lead to missed opportunities for organizational im-provement. When agency priorities and budgetary constraints get inthe way of implementing organizational change strategies based onsurvey findings, respondent time is wasted. Even those factors whichcould be addressed at minimal cost, such as improving organizationalcommunication and an emphasis on providing quality services, werenot addressed in this case, which may in turn impact morale. Whilethe findings regarding individual factors associated with intent to re-main employed are interesting as well as consistent generally withthe literature on this topic, review of regional variations in the studystate demonstrates that the diamonds may be in the details from an or-ganizational improvement standpoint for individual agencies.

The relationship between organizational and personal factors, andretention of staff has been well-documented (e.g. Zlotnick et al.,2005a,b) and this study supports many of these findings. The bivari-ate relationships identified demonstrate the importance of the roleof the range of organizational effectiveness constructs measured bythe SOE in retaining staff, in addition to emphasizing the potentialfor promoting retention through a positive organizational culture inwhich open and multi-directional communication flows. Withoutquestion, the significant influence of supervision (e.g. Ellett et al.,2006; Smith, 2004), and the role of organizational culture (e.g. Ellett& Millar, 2005, Landsman, 2001) supported by a body of literatureare echoed here.

As was mentioned in the introduction, these data were collectedin part for the purpose of psychometric comparison of similar instru-ments for a totally unrelated study. When the child welfare agencywas approached with the opportunity to collect data regarding orga-nizational factors and their relationship to staff retention at no cost,administrators readily agreed. Given resource limitations, child wel-fare agencies often cannot commit funding to such research, evenwhen the benefit of data-based decision-making is understood. Theagency was struggling with staff turnover, and the pressure of im-proving organizational, practice and client outcomes. The agency fa-cilitated data collection and encouraged staff participation. The

agency and researcher agreed upon a plan to bring together a teamof individuals on the state level to review study findings and deter-mine a preliminary course of action.

When the findings of this study were presented to agency adminis-trators and managers, they were met with a great deal of enthusiasm.Thefindingswere interpreted as being in linewith administrative prior-ities, and a great deal of discussion ensued regardingways in which thefactors within the control of the agencywithout significant fiscal alloca-tion. Agency staff were particularly interested on those constructswhich varied significantly based on geographic region, and urban/rural status. Plans were made to share the findings with regional man-agement for discussion of strategieswhich could bedeveloped on a geo-graphical basis to address unique needs. This approach is consistentwith the “organization-in-environment” perspective advocated byLandsman (2002, p.791) in which differences in constructs associatedwith child welfare practice and organizational approaches are viewedthrough the lens of rural versus urban constructs. An internal qualitativestudy involving regional focus groups was planned. A logical next stepwould have been to develop regional design teams to develop and im-plement organizational change strategies, such as has been describedby Strolin-Goltzman et al. (2009) in which county-level organizationalinterventions were implemented to address factors identified throughanalysis of organizational data to be associated with child welfare turn-over in New York State (Caringi et al., 2008). Unfortunately, the successrealized in these studies was not enjoyed in the current study becauseregional design teams were never developed.

Within the discussions among agency administrators and univer-sity researchers, an argument for annual measurement of organiza-tional effectiveness was made. Examples of other state child welfareagencies that have made the commitment to such measurement atannual intervals were provided (e.g. Collins-Camargo et al., 2009).Consideration was paid to the prospect of follow up study, and a relat-ed proposal was developed for an intervention that was designed toenhance frontline supervision and perceptions of organizational cul-ture in regions and local supervisory units. Unfortunately, the onsetof budgetary constraints and the impending change of upper manage-ment that is so typical of many public child welfare agencies pre-vented fulfillment of many of these plans in this particular state.

Sadly, this is a story that should sound very familiar to those work-ing in the field of child welfare. So often, these agencies are faced withdifficult decisions regarding how to use limited funds. The federalgovernment is asking states to create performance improvementplans regarding their failure to meet desired standards in client out-comes in addition to systemic factors like training and quality assur-ance based on the Child and Family Services Review Process(Administration for Children and Families, n.d.). In the professionalliterature, although there is more discussion of performance improve-ment and organizational change to achieve outcomes (e.g. Chaffin,2006; Lindsey & Schlonsky, 2008; Weigensberg, 2009), child welfarefunds are often not allocated to measure the extent to which thestate is making improvement in organizational factors like those mea-sured in the SOE. Were the state to make an investment in measure-ment of organizational factors over time, however, it would not onlyprovide the state with critical information on perceptions regardingaspects of organizational functioning which they have worked to im-prove, but it would model an evaluative, data-driven approach tochild welfare.

Since this study, the SOE has been updated into a new instrumententitled the Survey of Employee Engagement (SEE). Modificationswere made based on current organizational practice and theory, theexperience of those using the SOE over time, cultural adaptationand technology enhancements, and was totally unrelated to the prin-cipal component analyses conducted for the current study. The focusis expressed as “…the key drivers relative to the ability to engage em-ployees towards successfully fulfilling the vision and mission of theorganization.” (Institute for Organizational Excellence, 2009, p. 1).

294 C. Collins-Camargo et al. / Children and Youth Services Review 34 (2012) 289–295

The SEE is shorter than the SOE, and 14 constructs are now included.There is increased emphasis in the instrument on organizationalculture and climate, and the ability of organizations to comparefindings over time is enhanced. In its current form, the usefulness ofsuch an instrument to organizations wishing to standardize measure-ment longitudinally as improvement initiatives are implemented isenhanced. No doubt other instruments may serve a similar purpose.

Testa and Poertner (2010) advocate a results-oriented approach toaccountability that includes outcomes monitoring, data analysis, re-view of research, evaluation and quality improvement in the childwelfare setting. The professional literature describes increased inter-est in program accountability and evidence-based practice (EBP) inthe social services (e.g. Small, Cooney, & O'Connor, 2009; Thyer,2008). Franklin and Hopson (2007) noted the need for examinationof organizational and learning conditions that would promote the de-velopment of EBP, although others argued that achieving EBP may beelusive in a system like child welfare that is just beginning to trackoutcomes for the families and children it serves (Poertner et al.,2000). The literature has not documented a consistent use of EBP inchild welfare (Usher & Wildfire, 2003). Wulczyn (2005) questionedthe extent to which many child welfare practitioners know how touse data to improve their performance—the same may be said formany of the agencies in which they work.

Frequent shifts in child welfare leadership are common due to thepolitical nature of appointment to these positions. But given this reality,how can agencies sustain organizational improvement efforts overtime? How can agencies truly rise to the level of accountability beingcalled for in the literature (e.g. Waldfogel, 2000)? The federal Childand Family Services Review process sets the tone for long term assess-ment and improvement, however states which really want to takethis seriously should consider measuring organizational improvementover time.

This study documents how organizational effectiveness variablesare important correlates of and antecedents to employee retention.The correlations between Intent to Remain Employed and the twoSOE variables (.73, .62) are quite high for this kind of research,where r=.35 or .40 might be more typically considered rather high.Therefore, these results document the importance of the work placeor organizational variables as well as external environment variables(such as external communication and relationships) as these relate toemployee retention.

This study stops short of testing the extent to which increasingemployee retention through strengthening organizational effective-ness variables as measured by the SOE would result in improvedCFSR outcomes which would be an important area for further re-search. Glisson et al. (2006), who report on a study demonstratingthe relationship between culture and climate in child welfare andjuvenile justice agencies, and children's access to mental health ser-vices, note that studies linking the conceptual relationship betweenculture and climate to actual outcomes are rare. As agencies striveto improve these outcomes there are some variables which aremore amenable to change within the agency's locus of control.While the agency may not be able to control the severity of problemsexperienced by the children and families they serve, they can impactthe interpersonal work environment and organizational culture. Aprior study, for example, demonstrated statistically significant rela-tionships between frontline supervision, a professional organizationalculture promoting evidence-based practice, and worker self-efficacyin child welfare tasks (Collins-Camargo & Royse, 2010). In a very ex-ploratory way, a related study identified a relationship between thosesame variables and regional case level standards or outcomes such astimeliness of initial contact with families, percent of cases open over12 months and child maltreatment recurrence (Collins-Camargo,2005). The use of periodic measurement of organizational effective-ness and its relationship to client outcomes is a critical area for fur-ther research in the field.

This study also contributes to the research base for both the IREand the SOE, at least in its earlier form. The correlations reported be-tween the IRE and SOE also add to the criterion-related validity char-acteristics of both measures. The results of the current study suggestthat a multidimensional measure of organizational effectiveness alsocorrelates strongly and positively with IRE. This adds to the buildingevidence of the construct validity of the SOE.

There is a call for studying efforts to promote an organizationalculture that focuses on outcomes and evaluation in child welfareagencies (Carrilio, Packard, & Clapp, 2003; Webster, Needell, &Wildfire, 2002). Barth (2008) warned that the field must focus onbetter ways to achieve outcomes before such processes are imposedexternally by those who may not understand the system. Externallyimposed mandates are not uncommon in this field. There is a needfor further study of how measures of organizational improvementsuch as the use of the SOE by child welfare agencies, and subsequentdata-driven decision-making regarding improvement initiatives actu-ally effect the achievement of organizational, practice and clientoutcomes.

As was noted above, an example of how organizational percep-tions data can be analyzed in conjunction with actual client and com-munity indicators to assess relationships has been published usingthe SOE (Collins-Camargo & Royse, 2010). However, this is only thetip of the iceberg in terms of how such data could be used, particular-ly using multilevel modeling techniques such as Hierarchical LinearModeling (HLM) (Bryk & Raudenbush, 1992), which has been used,for example, to examine organizational culture and climate as a pre-dictor of access to mental health services (Glisson et al., 2006).Ellett et al. (2003) found the human caring variable of professionalcommitment was a strong (r=.73; pb .001) correlate of IRE forfront line workers with three years or less of employment in childwelfare. For the total sample of child welfare workers this correlationwas r=.67. It would seem that the complex relationships amongworker/personal variables, organizational variables, worker practice,and organizational (such as IRE) and client outcomes is an area inneed of ongoing research in child welfare.

The purpose of the current study was to test the hypothesis thatthere was a statistically significant positive relationship between sub-scales of the Survey of Organizational Excellence (SOE) and the Intentto Remain Employed (IRE) scales, and this hypothesis was stronglysupported. A more practice-oriented purpose was to provide infor-mation to the public child welfare agency that could assist them indeveloping and implementing strategies for improving organizationaleffectiveness and employee retention. While such information wasprovided and the researchers and child welfare administrators en-gaged in collaborative dialog regarding the use of these findings tar-geted strategies were left largely unimplemented. Ongoing study isneeded of such research-to-practice collaborations, and how the im-plementation of an evidence-informed approach to organizational in-terventions in child welfare can be related to organizational, frontlinepractice and client outcomes over time.

References

Administration for Children and Families (n.d.). General Findings from the Federal Childand Family Services Review. Retrieved June 29, 2010 from http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/genfindings04/genfindings04.pdf

Alexander, J. A., Lichtenstein, R., Oh, H. J., & Ullman, E. (1998). A causal model of volun-tary turnover among nursing personnel in long term psychiatric settings. Researchin Nursing & Health, 21, 415–427.

Anderson, N. R., & West, M. A. (1998). Measuring climate for work group innovation:Development and validation of the team climate inventory. Journal of OrganizationalBehavior, 19, 235–258.

Barth, R. P. (2008). The move to evidence-based practice: How well does it fit childwelfare services. Journal of Public Child Welfare, 2(2), 145–171.

Birleson, P. (1999). Turning child and adolescent mental-health services into learningorganizations. Clinical Child Psychology and Psychiatry, 4, 265–274.

Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models for social and beha-vioural research: Applications and data analysis methods. Newbury Park, CA: SagePublications.

295C. Collins-Camargo et al. / Children and Youth Services Review 34 (2012) 289–295

Burke, W. W., & Litwin, G. H. (1992). A causal model of organizational performance andchange. Journal of Management, 18(3), 523–545.

Caringi, J., Strolin-Goltzman, J., Lawson, H., McCarthy, M., Briar-Lawson, K., & Claiborne,N. (2008). Child welfare design teams: An intervention to improve workforce re-tention and facilitate organizational development. Research on Social Work Practice,18(6), 565–574.

Carrilio, T. E., Packard, T., & Clapp, J. D. (2003). Nothing in—nothing out: Barriers to theuse of performance data in social service programs. Administration in Social Work,27(4), 61–76.

Chaffin,M. (2006).Organizational culture andpractice epistemologies. Clinical Psychology:Science and Practice, 13(1), 90–93.

Cohen, B. J., & Austin, M. J. (1994). Organizational learning and change in a public childwelfare agency. Administration in Social Work, 18(1), 1994.

Collins-Camargo, C. (2005). A study of the relationship between effective supervision,organizational culture promoting evidence-based practice, worker self-efficacy, andoutcomes in public child welfare. Unpublished Doctoral Dissertation. Lexington,KY: University of Kentucky.

Collins-Camargo, C. (2007). Administering research and demonstration projects aimedat promoting evidence-based practice in child welfare: Challenges and rewards.Journ of Evidence-based Social Work, 4(3/4), 21–38.

Collins-Camargo, C., & Royse, D. (2010). A study of the relationships among effectivesupervision, organizational culture, and worker self-efficacy in public child wel-fare. Journal of Public Child Welfare, 4(1), 1–24.

Collins-Camargo, C., Sullivan, D., Washeck, B., Adams, J., & Sundet, P. (2009). Using amultifaceted clinical supervision initiative to improve recruitment, retention andpractice in the public child welfare agency. Child Welfare, 88(5), 87–107.

Cyphers, G. (2005). Report from the Child Welfare Workforce Survey: State and countydata and findings. Washington, DC: American Public Human Services Association.

Ellett, A.J. (2000). Human caring, self-efficacy beliefs, and professional organizationalculture correlates of employee retention in child welfare. Unpublished doctoraldissertation, Louisiana State University, Baton Rouge.

Ellett, A. J., Collins-Camargo, C., & Ellett, C. D. (2006). Personal and organizational cor-relates of outcomes in child welfare: Implications for supervision and continuingprofessional development. Professional Development: The International Journal ofContinuing, Social Work Education, 9(2–3), 44–53.

Ellett, A. J., Ellett, C. D., & Rugutt, J. K. (2003). A study of personal and organizational fac-tors contributing to employee retention and turnover in child welfare in Georgia. FinalProject Report. Athens, Georgia: School of Social Work, University of Georgia.

Ellett, A. J., & Millar, K. (2005). Professional organizational culture and retention inchild welfare: Implications for supervision and professional development. Profes-sional Development: The International Journal of Continuing Social Work Education,7(3), 30–38.

Franklin, C., & Hopson, L. M. (2007). Facilitating the use of evidence-based practice incommunity organizations. Journal of Social Work Education, 43(3), 377–404.

Glisson, C. (2007). Assessing and changing organizational culture and climate for effec-tive services. Research on Social Work Practice, 17(6), 736–747.

Glisson, C., Dukes, D., & Green, P. (2006). The effects of organizational culture and climateon the access to mental health care in child welfare and juvenile justice systems. Ad-ministration and Policy in Mental Health and Mental Health Services Research, 33(4),433–448, doi:10.1007/s10488-005-0016-0.

Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and interor-ganizational coordination on the quality of outcomes of children's service systems.Child Abuse & Neglect, 22(5), 401–421.

Graef, M. I., & Hill, E. L. (2000). Costing child protective services staff turnover. ChildWelfare, 79(5), 517–533.

Institute for Organizational Excellence (2009). The SOE is now the SEE.Austin, TX: Universityof Texas.

Landsman, M. J. (2001). Commitment to public child welfare. The Social Service Review,75(3), 386–419.

Landsman,M. J. (2002). Rural childwelfare practice froman organization-in-environmentperspective. Child Welfare, 81(5), 791–819.

Lauderdale, M. (1999). Reinventing Texas Government. Austin, TX: University of TexasPress.

Lauderdale, M. (2001). Michael Lauderdale: In response to Raymond Albert. Adminis-tration in Social Work, 26(1), 107–110.

Lindsey, D., & Schlonsky, A. (Eds.). (2008). Child welfare research: Advances for practiceand policy. New York: Oxford University Press.

Martin, J., & Siehl, C. (1983, Autumn). Organizational culture and counterculture: Anuneasy symbiosis. Organizational Dynamics, 52–64.

Mor Barak, M. E., Nissly, J. A., & Levin, A. (2001). Antecedents to retention and turnoveramong child welfare, social work, and other human service employees: What canwe learn from past research? A metanalysis. The Social Service Review, 75(4),625–661.

Morrison, T. (1997). Learning, training and change in child protection work: Towardsreflective organizations. Social Work Education, 16(2), 20–43.

Poertner, J., McDonald, T., & Murray, C. (2000). Child welfare outcomes revisited. Childrenand Youth Services Review, 22(9/10), 789–810.

Schein, E. H. (1985). Organizational culture and leadership. San Francisco: Jossey-Bass.Shim, M. (2010). Factors influencing child welfare employee's turnover: Focusing on

organizational culture and climate. Children and Youth Services Review, 32(6),847–856.

Small, S. A., Cooney, S. M., & O'Connor, C. (2009). Evidence-informed program improve-ment using principles of effectiveness to enhance the quality and impact of family-based prevention programs. Family Relations: Interdisciplinary Journal of AppliedFamily Studies, 58, 1–13.

Smith, B. D. (2004). Job retention in child welfare: Effects of perceived organizationalsupport, supervisor support, and intrinsic job value. Children and Youth Services Re-view, 27, 153–169.

Strolin-Goltzman, J., Lawrence, C., Auerback, C., Caringi, J., Claiborne, N., Lawson, H.,et al. (2009). Design teams: A promising organizational intervention for improvingturnover rates in the child welfare workforce. Child Welfare, 88(5), 149–168.

Testa, M., & Poertner, J. (Eds.). (2010). Fostering accountability: Using evidence to guildand improve child welfare policy. New York: Oxford University Press.

Thyer, B. (2008). The quest for evidence-based practice?: We are all positivists! Re-search on Social Work Practice, 18(4), 339–345.

United States General Accounting Office (2003). HHS could play a greater role in helpingchild welfare agencies recruit and retain staff. (Report No. GAO-03-357).Washington,DC: United States General Accounting Office.

United States General Accounting Office (2004). Child and family service reviews: Betteruse of data and improved guidance could enhance HHS's oversight of state perfor-mance (Report No. GAO04-333).Washington, DC: United States General AccountingOffice.

Usher, C. L., & Wildfire, J. B. (2003). Evidence-based practice in community-based childwelfare. Child Welfare, 72(5), 597–614.

Verbeke, W., Volgering, M., & Hessels, M. (1998). Exploring the conceptual expansionwithin the field of organizational behavior: Organizational climate and culture.Journal of Management Studies, 35(3), 303–329.

Waldfogel, J. (2000). Reforming child protective services. Child Welfare, 79(1), 43–58.Webster, D., Needell, B., &Wildfire, J. (2002). Data are your friends: Foster child welfare

agency self-evaluation in Los Angeles County with the Family to Family Initiative.Children and Youth Services Review, 24, 471–484.

Weigensberg, E.C. (2009). Child welfare agency performance: How are child, agency,and county factors related to achieving timely permanency outcomes for childrenin foster care? (Doctoral dissertation). The University of North Carolina at ChapelHill.

Wulczyn, F. (2005).Monitoring the performance of the childwelfare system. In RockefellerInstitute of Government (Ed.), Performance management in state and local government(pp. 54–62). Albany, NY: RIG.

Zlotnick, J., DePanfilis, D., Daining, C., & Lane, M. M. (2005). Retaining competent childwelfare workers: Lessons from research. Technical Report. Washington, DC: Institutefor the Advancement of Social Work Research.

Zlotnick, J., DePanfilis, D., Daining, C., & Lane, M. M. (2005). Understanding retention inchild welfare: Suggestions for further research and evaluation. Research Brief, 3.Washington, DC: Institute for the Advancement of Social Work Research.