Incorporating outcome measures into fire department

Running head: INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 1

Incorporating outcome measures into fire department management practices

Holger Durre

Boulder Fire-Rescue, Boulder, CO

INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 2

Certification Statement

I hereby certify that this paper constitutes my own product, that where the language of others is

set forth, quotation marks so indicate, and that appropriate credit is given where I have used the

language, ideas, expression, or writings of another.

Signed:

Date: 12/31/2018


Abstract

Governmental agencies have increasingly been called upon to be accountable to its

constituents (Han & Hong, 2016). One aspect of this has been the increasing use of outcome

measures in public agencies. The City of Boulder Fire-Rescue (BFR) department was

establishing a system to report outcome measures to improve operational efficiency and to

establish a foundation of data-driven decision making in the organization. The problem was that

the BFR had not identified an approach to incorporating outcome measures into its management

practices. The purpose of this research was to examine the incorporation of outcome measures

into the management practices of the fire service organization to establish options for BFR. The

research questions were (a) what are the best practices related to outcome measurement in the

public sector, (b) what are the factors associated with implementing outcome measures in fire

service organizations, and, (c) what approaches are fire departments utilizing to incorporate

outcome measures into management practices. The author used the descriptive research method

which included a review of the literature to establish current practice and the analysis of a survey

instrument.

The primary focus of the literature review was the exploration of systems-based

approaches to outcomes measurement in public sector organizations, particularly around the

logic model for strategic planning and outcome chain theory. Procedures included a survey to

identify current practices in the public sector and the fire service. The results of the research

identified several best practices for inclusion in BFR’s approach to integrating outcome measures

into its management practices. Recommendations included integrating scalable approaches to

output and outcome measurements as well as to integrate the logic model framework into

departmental business practices.


Table of Contents

Certification Statement ................................................................................................................... 2

Abstract ........................................................................................................................................... 3

Table of Contents ............................................................................................................................ 4

Introduction ..................................................................................................................................... 6

Background and Significance ......................................................................................................... 7

Literature Review.......................................................................................................................... 10

Best practices related to outcome measurement in the public sector ........................................ 12

Procedures ..................................................................................................................................... 15

Results ........................................................................................................................................... 18

Overview of Findings ................................................................................................................ 18

Detailed Results of Procedures .................................................................................................. 20

Discussion ..................................................................................................................................... 29

Recommendations ......................................................................................................................... 33

References ..................................................................................................................................... 36

Appendix A ................................................................................................................................... 40

List of Tables

Table 1 - Survey Question 1. Organization Type ....................................................................... 21

Table 2 - Survey Question 4. Aspect of Outcome Measurement Used ...................................... 22

Table 3 - Survey Question 5. Elements of incorporating outcome measures ............................. 23

Table 4 - Survey Question 9. Most influential factor contributing to outcome measurement .... 24

Table 5 - Survey question 10. Factors that proved challenging during implementation ............ 25


Table 6 - Survey Question 5. Factors related to the success of implementation ........................ 26

Table 7 - Survey question 12. Frequency of program performance discussion/reporting .......... 27

Table 8 - Survey Question 13. Statistical Validity by Program (n=144) .................................... 28

Table 9 - Survey question 14. Tools used to measure and report outcomes (n=144) ..................29


Incorporating outcome measures into fire department management practices

Introduction

One of the most effective ways for a fire department to communicate the relevancy of its

programs to its stakeholders is by objectively being able to demonstrate the impact on the public

it serves. However, despite the push for greater use of outcome measures, and its potential to

improve governmental operations, many municipalities have not yet widely adopted such

systems (De Lancer Jules & Holzer, 2002). This is important due to the increased reliance of

these approaches for funding justification as well as establishing the value of programs. One

example is the City of Baltimore, which has been using these approaches successfully for the last

15 years and subsequently has allowed the city “to fund high priority goals and to protect from

budget cuts the most effective services (“Baltimore case study,” 2018, para. 3).” Also, the same

case study found a direct positive correlation between the implementation of such systems and

the engagement of employees in public sector organizations as well as their general

understanding of organizational goals and priorities.

The City of Boulder and its fire department have been responsive to these developments

and have developed an innovation vision to improve the overall effectiveness of city services by

encouraging data-driven decision making and increasing operational efficiency. Also, the City of

Boulder Fire-Rescue (BFR) department is also pursuing accreditation through the Center for

Public Safety Excellence (CPSE) which requires the department to be able to objectively

demonstrate its ability to measure the effectiveness of its programs.

The problem is that BFR has not identified an approach to incorporate outcome measures

into its management practices. The purpose of this research is to examine the incorporation of

outcome measures into the management practices of fire service organizations to establish


options for the BFR to use in its approach. The research questions were (a) what are the best

practices related to outcome measurement in the public sector, (b) what are the factors associated

with implementing outcome measures in fire service organizations, and, (c) what approaches are

fire departments utilizing to incorporate outcome measures into management practices. The

author used the descriptive research method which included a review of the literature to establish

current practice and the analysis of a survey instrument.

Background and Significance

The City of Boulder Fire-Rescue Department (BFR) is a municipal fire department that

provides fire protection and emergency services to approximately 108,000 residents living in the

City of Boulder just northwest of the Denver Metropolitan Area (Boulder Economic Council,

2018). The BFR provides all-hazards response and prevention services to a service area that is

approximately 25 square miles in size. The agency is staffed by 117 uniformed personnel, 98 of

which staff seven fire stations. These personnel operate seven engine companies and one truck

company. The balance of the agency’s uniformed personnel, along with nine civilians, staff a

wildland division, training division, community risk division, and provide support and

administrative functions. The agency protects an estimated $21 billion in property (Boulder Fire-

Rescue [BFR], 2018, p. 43).

The City of Boulder is a hub for regional and national innovation as a result of a major

research university, several federal research labs, as well as a thriving entrepreneurial industry

being located in the city (Boulder Economic Council, n.d.). As an example, the city has been

home to the National Institute of Standards and Technology (NIST) since the 1960s which

created an influx of research scientists and a host of associated support industries (Engelke &


Paul, 2016). In addition, almost half of the members for Boulder’s city council have an

entrepreneurial or science-based background (City of Boulder Council website, n.d.).

In response to this environment, and at council direction, the City has embarked on an

innovation strategy to improve the overall effectiveness of city services. The overall architecture

of this strategy involves improving internal processes, becoming an incubator for ideas by

engaging community partners, and to amplify the impact of these initiatives by communicating

successes and progress ("City of Boulder Innovation Strategy," 2018). The city expects each

department to participate in achieving these initiatives and the BFR is beginning to contribute to

this process. To begin aligning with these initiatives the fire department budget has been

restructured to program-based budgeting. This budgeting method funnels funding to the program

level and forms a closer connection between the funding of a given service area and its

commensurate level of performance. To date, several improvements have been made in

measuring outputs, namely program activity levels in a more accurate way. In addition, the

department has hired a data and analytics project manager to assist with the implementation of

these initiatives.

However, the BFR does not currently have a cohesive approach to reporting outcomes of

the department’s program activity measures. This makes it difficult to connect the previously

mentioned program funding and related activity measures to outcomes that can objectively relate

the impact of a program on achieving the goals of the department. To connect these activities, the

BFR is in need of adapting its approach to performance management tracking to include the

ability to report the impact of the various programs it is responsible for administering. Without

the ability to tie the outputs and activities of programs directly to the impact that a program is


having, the department will not be able to realize the potential of progrtam-based budgeting fully

and will likely be at a funding disadvantage with other departments in the city.

Also, the BFR is currently an applicant agency under the Center for Public Safety

Excellence’s (CPSE) model of accreditation. Under the current model, agencies are required to

report the impact of program performance and do so on a regular basis. Performance Indicator

2D.2 is an example of this requirement as an applicant for accreditation must “continuously

monitor, assess, and internally report, at least quarterly, on the ability of the existing delivery

system to meet expected outcomes and identifies the remedial actions most in need of attention

(Center for Public Safety Excellence, 2015, para. 2D.2).” As written, a system for tracking these

outcomes does not currently exist at BFR, which creates a significant gap the agency must

overcome to be eligible for accredited status. This facet of accreditation intends to encourage

agencies to instill continuous improvement parameters into its operations in a sustainable

fashion. This would allow BFR to become a more resilient and sustainable agency. Therefore,

establishing an approach for integrating outcome measurement would allow the BFR to more

accurately track the impact of its programs, ensure a focus on continuous improvement and to

increase employee engagement by encouraging participation in these roles. The organization has

lacked an approach to implementing such practices, and as presented here the City needs to

overcome these limitations.

The BFR has encountered barriers to implementing outcome measures for a variety of

reasons. One reason is a lack of program-based tracking mechanisms that are focused on reliable

and reproducible data. This is in part related to the department not having tracked such measures

in the past and lacking the foundation to reduce the impediments to ease the access to such data.


The BFR’s goal in integrating outcome measures is therefore not only to report program impact

more reliably but also to improve the data tracking foundations on which these systems are built.

This applied research project (ARP) is directly linked to Unit 3: Thinking Systematically

as presented in the National Fire Academy’s RO125 Executive Leadership course, which states

as a terminal objective to “examine the systems within which the adaptive challenge exists, using

purposeful collection of data to help clarify and define what occurs within these systems. (United

States Fire Administration, 2015, p. SM 3-1).” This ARP is related to this curriculum by

examining the potential for systems of integrating outcome measures at the BFR and identifying

potential organizational change implications involved with the establishment of these practices.

Therefore, this paper uses the content areas of the R0125 course as the basis of examining and

evaluating the options available to the BFR for integrating these systems.

This ARP is relevant to United States Fire Administration (USFA) Goal One which is to

“reduce fire and life safety risk through preparedness, prevention, and mitigation. (United States

Fire Administration [USFA], 2014, p. 12).” This applied research project will directly support

this goal by helping the BFR become a more outcomes-oriented organization which will directly

reduce the commensurate risks in the City of Boulder.

Literature Review

The literature review sought to identify the current best practices related to outcome

measures in the public sector as well as to identify those areas in which fire service organizations

have adopted them into their operations. These areas are later explored within the research

questions of this paper. The review identified the major components relevant to outcome

measurement systems and provided context on how these can be applied in organizations. This


foundation of academic and industry literature supported the assembly of a survey that is

discussed later in this paper.

The primary focus of the literature review was the exploration of systems-based

approaches to outcomes measurement in public sector organizations. Much of this work revolved

around the logic model for strategic planning and outcome chain theory. The former being a

systematic and linear approach linked to establishing clear problem statements and the former

being focused on how outcomes are the product of a confluence of events and must, therefore, be

examined more broadly as often, not a single input itself can be attributed to the achievement of

a specific result.

To begin, it is important to establish the meaning of outcomes for the context of this

paper. In this case, outcomes will be referred to as the measurable impact of program activities

(Joly, Polyak, Davis, & Brewster, 2007). Lee, Johnson, and Joyce (2004, p. 111) posit further

that “when a government service has affected individuals, institutions [or] the environment, an

impact has occurred.” This paper will focus on specifically establishing those impacts that can be

reported to stakeholders. Performance management practices are not new to the public sector but

have been slow to evolve. Based on the work of Edward Deming, the roots of these systems are

found in the total quality management (TQM) approaches that found its way into government

and has influenced budgeting methods in this sense for many years (Lee et al., 2004). The roots

of these efforts began in the federal government under the Hoover Commission that first

established the foundation for performance budgeting (Milakovich & Gordon, 2004). These

efforts evolved into a broader discussion of budgeting as a central facet of performance

management in place today. Local governments have increasingly adopted performance

measurement approaches to its budgeting processes (Newcomer, 2007). These movements are all


intended to increase the ability of governments to report the direct impact of its activities to the

citizens that ultimately fund program activities.

Best practices related to outcome measurement in the public sector

Two primary methods were uncovered by the literature review in use today by public-

sector organizations currently integrating these into management practices. The first is the logic

model for strategic planning and the second is outcome chain theory. These two approaches each

tie program inputs and connect them to the various activities with the ability to report the outputs

as well as related outcomes. These will be explored separately to establish a baseline for the

assembly of the survey instrument.

The logic model for strategic planning “describes the logical linkages among program

resources, activities, outputs, customers reached, and short, intermediate and longer-term

outcomes (McLaughlin & Jordan, 1999, p. 65).” The system relies on the clear establishment of

a problem statement and associated goals that relate to solving these. Program inputs are clearly

defined which can be funding, people, or physical resources (Baxter, Blank, Buckley-Woods,

Payne, & Rimmer, 2014). These authors established in particular the importance of measuring

program outputs that are connected to goal statements developed during the establishment of a

program. These methods are intended to “make an explicit, often visual, statement of the

activities that will bring about change and the results you expect to see for the community and its

people (University of Kansas, 2018, para. 1).” Specifically, public sector organizations are

advised to take into consideration organizational context and complexity when establishing such

models for outcomes measurement (Heinrich, 2002). This is particularly true because the various

components identified earlier need to be assessed individually to ensure that the overall

organization can implement them successfully. The literature identified the benefits of using


logic models in helping an organization keep a focus on outcomes, building an understanding of

program accomplishments, providing a common language that focuses work, and enhancing

prioritization and allocation of resources (Taylor-Powell & Henert, 2008). These findings

influenced the research by identifying the specific components that were used during the

assembly of the survey instrument described in the procedures. The specific components of the

logic model framework were extracted for use in the survey instrument, especially outcome or

goal statement development, identifying program inputs, measuring reliable activity measures,

and developing key performance indicators.

The second major approach found by the literature review was the use of outcome chains.

ThiOutcome chains are also described as outcomes mapping, and in either case, involves the

systematic representation of the process of achieving outcomes in a manner that can be mapped

in its subcomponents. One key distinguishing characteristic of outcome mapping or outcome

chains is that the model can acknowledge that an individual program in and of itself may not be

solely for achieving a given output. In this case “a program is not claiming the achievement of

development impacts; rather, the focus is on its contributions to outcomes. These outcomes, in

turn, enhance the possibility of development impacts — but the relationship is not necessarily a

direct one of cause and effect (Earl, Carden, & Smutylo, 2004, p. 1).” Another example of this in

outcome chains. Like in the outcomes mapping example above, outcomes, or result chains,

“provide a theoretical model for defining the interrelated components of a project that are

required for its success (Parsons, Gokey, & Thornton, 2013, p. 7).” The primary feature then of

this approach is that multiple programs or factors can influence the outcome of a certain

program. This is a key distinguishing factor of outcome chains and outcome mapping when

compared to the logic model. “As such, outcome mapping assesses a program holistically and is


based on the premise that a program needs to know not only about development results, but also

about the processes by which these result were attained, and about its internal effectiveness (Earl

& Carden, 2002, p. 4).” A prominent place that these methods are currently being used is in

development work such as international outreach programs and community development. Given

that the fire service provides a service to the community that often attempts to reach across

boundaries of socioeconomics and to provide education, these models could be potentially

valuable for integrating outcomes into the management practices of a fire department.

One key aspects of outcome mapping and outcome chains that the literature identified are

that these are not always the best to use for smaller and contained programs (Earl et al., 2004).

This is due to the nature of outcome chains looking at the comprehensive impact of those

activities not under the direct control of a certain program. In other words, the activity of

complementary programs is taken into consideration which significantly increases the

complexity of the measurement activities around the program but also influences the ability of

smaller organizations to implement sophisticated enough measurement systems that can account

of the impact of those external factors sufficiently to be effective. For example, the

implementation of an outcome mapping approach in a community fish rearing program was

evaluated and this fond that “the inherent non-linear complexity of the development process and

the difficulties of monitoring impact and assigning attribution” was a limiting factor of the

implementation of the program (Sheriff & Schuetz, 2010, p. 4). These findings influenced this

research in describing the need for an organization to identify the ability to implement such a

complex system of assessing these factors adequately. Fire departments already deal with the

impact of incomplete internal data and expanding this problem to external factors may be a

limiting factor use in a medium-size urban fire department.


Finally, the literature review identified key factors related to staffing and deployment of

an outcomes-based measuring system (Pearson, 2005). Pearson indicates that the availability of

support staff, to include those dedicated to IT are crucial in ensuring these systems are

implemented effectively. This has been found in other cases as well such as the implementation

of program performance measurement systems (Iversen & Ngwenjama, 2006). A related

response to this has been the fire service’s recent increase in dedicated staff for performance

management related to accreditation and program-based performance measurement. These are

also supported as best practices in general fire service administration (Bruegman & Lindsey,

2009). This influenced this research to ensure that these factors were assessed in the research and

assembly of the survey instrument.

Procedures

The purpose of this research was to examine the incorporation of outcome measures into

the management practices of fire service organizations to establish options for the City of

Boulder Fire Department. To perform this analysis, a survey instrument was administered to both

public sector organizations with a specific portion focused on fire departments. The survey was

designed by using the literature review of this paper as a foundation to examine current best

practices in the industry. The procedures sought to answer the research questions identified.

The research questions asked (a) what are the best practices related to outcome

measurement in the public sector, (b) what are the factors associated with implementing outcome

measures in fire service organizations, and, (c) what approaches are fire departments utilizing to

incorporate outcome measures into management practices? The results are discussed in the

appropriate sections of this paper, and any applicable survey instruments are presented in the

appendices of this paper.


The survey was 14 questions in length and is represented in Appendix A of this paper.

The first three questions, as well as question 8, involved demographic background information

that was used to be able to categorize the responses and discern features based on agency type

and the level of application of outcomes-based practices in use by the particular agency.

Question 4 through 8 were posed to all participants and asked general questions that were aimed

at assessing general aspects of outcomes measurement in both public sector and fire service

entities. Questions 9 through 11 assessed what factors were associated with implementing

outcome measures in fire service organizations. Finally, questions 12 through 14 assessed the

specific approaches fire departments were utilizing to incorporate outcome measures into their

management practices.

The survey was distributed using the International Association of Fire Chief’s

KnowledgeNet, the Rocky Mountain Accreditation and Professional Credentialing Consortium

website as well as the LinkedIn professional networking application. Finally, the survey was

posted to the International City and County Manager’s Association (ICMA) performance

management discussion board. The survey was held open for 28 days on the Survey Monkey

platform. During this time, a total of 246 responses were received, and the overall completion

rate of the survey was 60%.

To address the first research question (a) what are the best practices related to outcome

measurement in the public sector, the survey instrument used the specific findings of the

literature review related to the outcome practices that were identified that the academic literature

identified as best practices for public sector organizations. The specific practices are represented

in question 4. Question 5 was focused on assessing the factors related to incorporating these

practices into the management practices of a specific organization. These were also obtained


from the literature review. During the literature review, staffing and budgeting were identified as

crucial factors related to the success of outcomes integration and, therefore, questions 6 and 7

assessed the level of these factors.

To answer the second research question (b) what are the factors associated with

implementing outcome measures in fire service organizations, question 8 determined if the

respondent was a member of a fire service organization. Questions 9 and 11 assessed directly

those items identified in the literature review that were correlated to the success of these

programs in the academic literature. The literature review identified barriers to implementation,

and these were assessed in survey question 10. The specific participation rates and applicable

results are presented in subsequent sections of this paper.

The third and final research question asked (c) what approaches are fire departments

utilizing to incorporate outcome measures into management practices? Once again, the survey

instrument was used as described for the previous research questions. In this case, questions 12

through 14 were related to how fire service organizations operationalized these practices.

Specifically, participants were asked to report on their activity in this area related to the

frequency of reporting out on the status of program activity, if these results were statistic valid,

and finally, what specific tools were used during the implementation of the processes. The last

question was primarily intended to identify potential tools for Boulder Fire-Rescue (BFR) to

consider implementing. The questions for this part of the survey were designed based on the

author’s knowledge of BFR administrative operations.

These procedures and the study itself was impacted by some limitations that are

important to discuss. The first of these was the lack of participation by non-fire service public

sector organizations. Despite several attempts to distribute the survey to distribution lists that


were generally focused on the public sector, such as the ICMA discussion boards, very few

responses were received. Therefore, the results are similarly affected and represent mainly the

experience of fire-service organizations. While this is still valuable as these are a largely a part of

the public sector, the scope of the results should be interpreted with this in mind.

Secondly, the academic literature was often broader than government service

organizations. Much of the literature review uncovered general practices and those in use by the

medical profession for patient management. While these still applied, the paper could have

benefited for more specificity related to the experience in the public sector.

Results

Overview of Findings

The original research posed three distinct questions. These were (a) what are the best

practices related to outcome measurement in the public sector, (b) what are the factors associated

with implementing outcome measures in fire service organizations, and, (c) what approaches are

fire departments utilizing to incorporate outcome measures into management practices? To

answer these questions, the results of a survey administered to both public-sector and fire service

entities were evaluated.

The first research question, aided by the literature review, discovered that best practices

in outcome measurement currently revolves around the implementation of a systematic approach

to outcome measurements such as those found in the logic model for strategic planning and

outcome chain or outcome mapping approaches. The survey indicated that the industry was

utilizing primarily measures of program activity in the form of outputs and activity (53.57%),

followed by outcome and goal statement development (36.90%), and the use of key performance

indicators (36.90%). This portion of the research also attempted to identify the most important


elements involved in incorporating outcome measures. The respondents indicated the three most

important elements were establishing clear goals to define program success (75.6%), clearly

identifying the problem being addressed by a program/activity (72.62%), and establishing short,

medium, and long-term activity and output measures (56.55%). Two key findings in research

question one were that 86.69% of respondents indicated they did not have staff dedicated to

manage their outcome measurement system and that only 33.33% connected outcome measures

to their budgeting process. This stands in contrast with the finding of the literature review that

these practices are encouraged. These results indicate the specific components in use by agencies

mirror those in the literature, although only some components are being used consistently.

The second research question attempted to answer what factors were associated with fire

departments during the implementation of outcome measurements. The survey assessed what

factors led to a department implementing such measures and the research found that this was

related to improving programs (42.28%), protecting funding or budget defense (27.52%), and

continuous improvement efforts such as accreditation (15.44%). The respondents indicated that

the most challenging aspects were a lack of organizational buy-in (27.52%), a lack of staffing to

support these functions (26.85%), and a lack of financial support (20.81%). One interesting

finding here was that very few agencies (6.71%) indicated that they felt that a lack of technology

was a limiting factor. Based on these results it was not surprising that the most critical factor

identified in ensuring the success of an outcomes measurement system was communicating the

value of the program (37.58%), clearly identifying the value of the program (37.58%) and

removing barriers to implementation (23.49%). These can be seen as directly linked to creating

buy-in as asked by the previous question.


To answer the third question, which assessed what approaches departments were using,

respondents were asked about the frequency of reporting activities, statistic validity, and the use

of measurement tools in the process. Respondents indicated that the reporting of progress and

outcomes measures is taking place at either monthly (38.89%) or quarterly (33.33%) intervals.

The remainder of agencies (27.78%) were reporting at either daily, weekly, or annual intervals.

The survey also asked if respondents felt that they could report statistically valid outcome

measures and if so, in which program areas this was being reported in. One interesting finding

here was that 93.75% of the respondents indicated they could do so in emergency response,

which is a very high percentage. A further 70.83% could do so in training, and 59.03% could do

so in code enforcement. However, the fewest respondents could report these in facilities

(15.97%) and fleet (36.11%). The question did not define statistic validity, and further research

would need to determine the reliability of these responses. Finally, the last question assessed the

tools or the manner in which, agencies were reporting outcome measures in their management

practices. These were mainly program performance reports (68.75%), data visualization

(42.36%), and statistical analysis software tools (36.11%).

Detailed Results of Procedures

To answer the first research question, a survey was issued to answer what the best

practices related to outcomes measurement were in the public sector. The survey was built using

results from the literature review. The survey was open for a total of 28 days and received a total

of 246 responses. The survey consisted of three distinct sections that each aimed to answer one

of the research questions. The survey assessed demographics information in questions 1 through

3 and in question 8. The latter was the determinant of whether or not the respondent was a

member of a fire service organization. Question 1 ascertained the make-up of the organization


regarding focus. The primary demographic of the respondents represented local government with

a total of 236 (95.93%) of the 264 respondents reporting this as their background. A far distant

second was non-governmental organizations at 2.44% or 6 of the respondents. The results of the

questions are presented in Table 1.

Table 1

Survey Question 1. Organization Type (n=264)

Rank Agency Type Count Percent of Total

1 Local Government 236 95.93%

2 Non-Governmental Organization 6 2.44%

3 State Government 2 0.81%

4 Federal Government 2 0.81%

5 Educational Institution 0 0.00%

Survey questions 2 and 3 sought to gain an understanding of how respondents were

utilizing outcomes in decision making. While these questions were not directly designed to aid in

answering research question 1, it served as demographic background. Of the 246 respondents,

39.84% (98) of the respondents reported that they were using outcomes to establish programs

while only 15.58% (39) reported having discontinued a program based on the lack of clear

outcome measures.

Survey question 4 assessed specifically which of the best practices identified in the

literature review that a particular entity was using in their outcome measurement systems. While

all of the practices are displayed in Table 2, the three primary practices identified were

measuring the level of program activity through outputs with 53.57% (90) respondents indicating

they were tracking these. Also, 36.90% (62) of the respondents reported developing outcome


goal statements. The same number of respondents reported the use of key performance

indicators. Of note was that respondents seemed to be using components of the systems based

best practices identified in the literature review, but only 2.89% (5) reported using outcome

chains, while 1.79% (3) used the logic model.

Table 2

Survey Question 4. Aspect of Outcome Measurement Used (n=168)

Rank Outcome Measure Count Percent of Total

1 Measuring Outputs 90 53.57%

2 Outcome or goals statement development 62 36.90%

3 Key Performance Indicators 62 36.90%

4 Dashboards 49 29.17%

5 Analytical software 44 26.19%

6 Defining inputs 34 20.24%

7 None of the above 33 19.64%

8 Reliability indicators 19 11.31%

9 Outcome chains 5 2.98%

10 The Logic Model 3 1.79%

To assess what respondents felt were the most important elements in their outcome

measurement system, 75.62% (125) of the respondents reported that establishing clear goals to

define program/activity success was key in their efforts while 72.62% (122) reported that clearly

identifying the problem being addressed by a program was vital. 56.55% (95) stated that

establishing short, medium, and long-term activity and output measures were important.


Table 3

Survey Question 5. Elements of incorporating outcome measures (n=168)

Rank Element Count Percent of Total

1 Establishing clear goals 127 75.62%

2 Clear identification of problem being

addressed

122 72.62%

3 Establishing short, medium, and long-term

outputs

95 56.55%

4 Establishing a link between outputs and

outcomes

62 36.90%

5 Dedicated time to report progress 52 30.95%

6 Accurately identifying inputs 33 19.64%

The literature review had identified the importance of dedicating support staff to

outcomes measurement as well as the fact that connecting these systems to the budgeting process

was important. In question 6, respondents reported that only 11.31% (19) had dedicated staff,

while 33.33% (56) reported a connection to the budget process. These two survey questions

concluded the questions designed to answer the first research question (a) what are the best

practices related to outcome measurement in the public sector?

These results indicate that while logic models and outcome chains are seen as best

practices in the literature, the industry is currently only using elements of each as indicated in

survey question 4. Output measurement is prevalent in organizations, which is not a surprising

result, but the use of these systems was only reported in a small number of cases. Also,

respondents indicated that clear goal statement development and understanding the problem each


program is addressing was seen as some other best practices to use in the deployment of such

efforts.

To answer the second research question, the same survey was utilized to determine the

factors associated with implementing outcome measures in fire service organizations. Question 8

was a demographics related question that ascertained whether or not a respondent represented a

fire service organization. As mentioned in the procedures, only 1.76% (3) of the respondents

indicated that they did not represent such an organization. This was a very significant result that

will be later explored in the discussion section of this paper. While this was a limitation to gain a

better understanding of public sector best practices, the results for the remaining research

questions were supported by a high number of responses from those in the fire service.

Survey question 9 was aimed at determining the most influential factor that led to a

particular fire service organization to start measuring outcomes. This was important to ascertain

as background for BFR to determine if the motivations of the organization were aligned or

differed from the general experience of the population. 42.28% (63) of respondents indicated that

their primary factor was to improve programs objectively, while 27.52% (41) reported the need

to protect funding with these efforts. A further 15.44% (23) stated that other continuous

improvement efforts motivated their deployment of these methods. Of note was that only 4.03%

(6) of the respondents reported that pressure from elected officials contributed to their pursuit of

such initiatives. Table 4 represents these results in full.

Table 4

Survey Question 9. Most influential factor contributing to outcome measurement (n=149)

Rank Factor Count Percent of Total

1 Improving Programs 63 42.28%


2 Budget Defense 41 27.52%

3 Related to other improvement efforts 23 15.44%

4 Meeting community expectations 16 10.74%

5 Political pressure 6 4.03%

6 Industry pressure 0 0.00%

Question 10 was designed to ascertain the challenges fire service organizations faced

with the implementation of these efforts. 27.52% (n=41) reported a lack of buy-in and 26.85%

(n=40) stated that a lack of staffing contributed to their challenges. Finally, only 6.71% (n=10)

stated that a lack of technology was a challenge for these agencies.

Table 5

Survey question 10. Factors that proved challenging during implementation (n=149)


1 Lack of buy-in 41 27.52%

2 Lack of staffing 40 26.85%

3 Lack of financial support 31 20.81%

4 Lack of expertise 37 18.12%

5 Lack of technology 10 6.71%

The final question designed to address the second research question was related to what

factors were most contributory to the success of an agency’s efforts in outcome measurement.

37.58% (56) of the respondents reported that communicating the value of the outcome

measurement program was vital in ensuring overall program success. A further 28.86% (43)

reported that clearly identifying the parameters and intent of a program was key, along with


23.49% (35) of the respondents that indicated that removing barriers to implementation was most

related to the success of a program. On the other hand, only 1.34% (2) of the respondents

reported that celebrating success was key which contrasts most organizational change strategies.

Table 6 contains the complete results of this question.

Table 6

Survey Question 5. Factors related to the success of implementation (n=149)


1 Communicating the value of the program 56 37.58%

2 Clearly identifying the parameters and intent of the

program

43 28.86%

3 Removing barriers to implementation 35 23.49%

4 Building an implementation team 10 6.71%

5 Dealing with setbacks 3 2.01%

6 Celebrating success 2 1.34%

These results, therefore, help to answer the second research question, which was to

determine the factors that were associated with implementing outcome measures in fire service

organizations. Respondents indicated that most initiated such efforts to improve programs and to

defend their budgets. These factors would be important at the BFR as well for consideration. The

surveys also revealed that the factors that most influenced these implementation efforts

negatively were a lack of buy-in, staffing, and financial support. These are also factors to

consider at BFR and will be discussed in greater detail in a later section. Finally, most agencies

reported that success was most connected to communicating the value of a program through

these initiatives, as well as clearly identifying the parameters of the program itself. Therefore, the


factors that are associated with implementing outcome measures in the fires service have been

identified in this sense and will be further discussed in the discussion section of this paper.

The final research question assessed what approaches fire departments are utilizing to

incorporate outcome measures into management practices. This was designed to determine the

specific frequencies, statistical grounding, as well the specific tools in use by departments. These

were assessed to determine if these would be valuable for inclusion at BFR in its effort to

establish a similar program. The survey used for the previous questions was utilized once again,

and the results were scored in the same manner as the previous components of the survey.

Survey question 12 examined the frequency at which respondents were reporting results

and discussing them during regular management activities. 38.89% (56) of the respondents stated

that they did so on a monthly basis, while 33.33% (48) stated that this was occurring quarterly.

The remainder of the respondents stated that they did so weekly, annually, or daily. These results

indicate that while some periodic examination of the progress of these measures is key, an

agency should determine its own approach based on internal restrictions and factors.

Table 7

Survey question 12. Frequency of program performance discussion/reporting (n=144)

Rank Frequency Count Percent of Total

1 Monthly 56 38.89%

2 Quarterly 48 33.33%

3 Weekly 24 16.67%

4 Annually 21 14.58%

5 Daily 4 2.78%


Survey question 13 assessed in which program areas an agency felt it could report

statistically valid outcome measures. This was meant to be a background question to determine

the state of sophistication of these efforts in the industry. The majority, 93.75% (135), of

respondents indicated that emergency response was an area that they could report this level of

outcome, while training at 70.83% (102) and code enforcement at 59.03% (85) was reported as

the third most common area of statistically valid outcomes. Of note is that no specific definition

was given of statistical validity and the respondent's interpretation of this may influence these

results.

Table 8

Survey Question 13. Statistical Validity by Program (n=144)

Rank Program Count Percent of Total

1 Emergency Response 135 93.75%

2 Training 102 70.83%

3 Code Enforcement 85 59.03%

4 Public Education 74 51.93%

5 Fleet 52 36.11%

6 Recruitment/Diversity 36 25.00%

7 Firefighter Wellness 34 23.61%

8 Facilities 23 15.97%

The final question assessed what tools agencies were using to report outcome measures in

their management practices. This was asked to ascertain the most common uses of tools by the

respondents and to understand what BFR should consider regarding implementing practices

related to deploying such a system. Most of the respondents, 68.75% (99) were using program


performance reports, while 42.36% (61) were employing data visualization methods. A further

36.11% (52) reported the use of statistical analysis software. Finally, these respondents reported

the use of meeting minutes in 31.25% (45) of the cases.

Table 9

Survey question 14. Tools used to measure and report outcomes (n=144)

Rank Tools Used Count Percent of Total

1 Program performance reports 99 68.75%

2 Data visualization 61 42.36%

3 Statistical analysis software 52 36.11%

4 Meeting minutes 45 31.25%

These survey questions related to research question three sought to answer what

approaches fire departments were using to incorporate outcome measures into their management

practices. It was clear that most agencies were utilizing either monthly or quarterly checkpoints

to ensure they stayed on track with their outcome measurement efforts and that most agencies

were utilizing program performance reporting and data visualization to report the progress of

these efforts.

Discussion

The research questions originally posed aimed to examine the incorporation of outcome

measures into the management practices of fire service organizations to establish options for the

City of Boulder Fire Department. The purpose was accomplished by carrying out the procedures

as outlined earlier which included a survey instrument administered to answer the three research

questions. To examine the implications for the BFR to integrate outcome measures into


management practices of the organization, these three questions will be examined to study the

relationship between the results of the survey and the literature review presented earlier.

The first research question attempted to answer what the best practices were related to

outcome measurement in the public sector. The literature identified two primary systems of

outcome measurement which included outcome chains as well as the logic model. Both of these

systems set out to establish an approach that took into consideration outcomes as a system of

components that interact with one another to create intended impacts (Earl et al., 2004). These

included the establishment of program parameters and goals, inputs, goal statements, as well

clearly defined impact endpoints that could be identified to ensure program success (Taylor-

Powell & Henert, 2008). This was contrasted in the findings of the administered survey that

revealed that while the various components of each of these systems were being utilized by the

public sector, as well as the fire service, not all the components were being used in conjunction

with one another.

This implied that BFR should consider evaluating the full implementation of such an

approach to attain the best practice scenario as identified in the literature review. The literature

review indicated that there is an inherent complexity associated with outcome measurement and

that these complexities must be considered in the implementation of such systems (Heinrich,

2002). Therefore, BFR should fully study the implications of implementing all the components

of such a system during the initial phases of such an effort to ensure that the various components

of the outcomes measurement system can be connected with one another effectively. This has

significant organizational implications as; currently, not all of the IT systems of BFR are

prepared to manage such an integration due to a reliance on legacy systems that have been in

place before the introduction of such measures. In addition, the literature indicated that outcome


chains are relatively complex to manage as they require the ability of an organization to track the

impact of external influences on outcomes (Earl & Carden, 2002). The BFR does not have the

ability to implement such sophisticated measurement systems and should consider this in the

selection of an approach.

Another significant finding in analyzing the results of the first research question was the

use of key performance measures (KPIs). These were previously not discussed in great length

but bear some significance concerning BFR’s implementation efforts. The third most utilized

component of outcome measures by the survey participants was the use of KPIs, and the

literature review indicated the general benefit of using such an approach to drive organizational

outcome focused results (Spekle & Verbeeten, 2014). Therefore, BFR should evaluate more fully

the use of KPI’s. The literature suggests that the primary benefit of using KPI’s is their tendency

to focus organizational energy toward the achievement of such measures due to the relative ease

of interpreting such metrics. This will involve an expansion of the current efforts underway to

create dashboards that are related to outcome achievement and will require an investment in the

information technology areas of the organization. Since BFR is in the initial stages of its

implementation, this should be considered in the integration process.

The second research question attempted to examine the specific factors associated with

implementing outcome measures in fire service organizations. The intent was to identify the

specific factors involved with such efforts to learn if there were considerations the organization

should consider before engaging in this effort. The most significant finding in this regard was

that the importance of communicating the value of outcome measurement, along with clearly

communicating the value of the program being measured, were key to such an implementation.

This is supported in the literature, particularly in the concept of return on investment. The idea


that the value of the program relates to an improvement of the standing of the organization as a

whole has been linked to the success of such implementation efforts (Phillips, 2011). This has

clear implications on BFR as this should be a central component of the implementation effort of

the new system of outcomes measurement in the department.

Another finding of the research that was related to this research question concerned

barriers to implementation. The survey participants indicated that a lack of buy-in was a barrier

that should be considered. While this was not directly found in the literature, general change

management practices indicate that this must be considered as a countermeasure. These findings

are supported by Kotter (2012) who posits that gaining buy-in for organizational change efforts

is a key to the overall success of change initiatives and new program implementation. This has

been important for BFR as a new administration has taken steps to alter the course of the

organization significantly. The author has recognized buy-in as being key to the success of any

change initiative in this organization.

The third research question attempted to answer what specific approaches fire

departments were utilizing to incorporate outcome measures into their management practices.

The survey participants were asked questions focused on reporting frequency, statistic validity,

as well as reporting tools in use by the agency being surveyed. The primary finding related to

answering this research question was related to the reporting frequency and manner in which the

results were reported. Most agencies were reporting monthly in the form of program reports that

were sent to a stakeholder such as a council. These findings are supported by the literature

review and the work of others. Particularly, the reporting to external stakeholders was seen as

vital in the literature (Thomas, 2006).


This finding is of importance to the BFR in its efforts as the current system of reporting is

already based on a streamlined process of monthly reporting of metrics through the use of a

community dashboard. However, the organization should make efforts to expand these initiatives

based on the findings of the research as well as the literature which indicate that external

accountability is directly related to the success of such initiatives. The regular reporting of

progress related to program performance should, therefore, be included in the design of such a

system.

A final finding of the research indicated that dedicated staffing was important in the

successful deployment of outcomes measurement systems. While BFR currently lacks a modern

information technology infrastructure, the staffing of a dedicated project manager for data and

analytics is not only relevant but should be a key change agent in this effort. While the findings

of the research indicated that most agencies were focused on emergency response, a key focus of

BFR is community risk reduction efforts, and as such, these will be a central component of the

new system of performance reporting at BFR.

Recommendations

Based on the results of the stated research questions several valuable recommendations

are being made for BFR to pursue during the implementation of its outcomes measurement

practices. These recommendations are based on the results of the survey administered and the

literature review as discussed. These recommendations are meant to help BFR design an

outcomes measurement system that integrates into the management practices of the organization,

which was the original purpose of the research.

It is recommended that the City of Boulder utilizes the logic model to integrate outcome

measures into the management practices of the organization. This integration should clearly


identify the problems each program is meant to address and develop commensurate goals for

solving these problems. To support these efforts, output measures that relate directly to short,

medium, and long-term outcome measures should be established. These recommendations are

based on the findings of surveyed organizations that indicate that these factors most contribute to

the success of such efforts.

It is also recommended that BFR clearly communicates the intended outcome of these

systems to program managers and to incorporate these into quarterly staff meetings that are used

to report the progress of these efforts. The BFR has an advantage in the implementation of these

systems as it currently employs a full-time data and analytics project manager that will need to

be the primary support for these efforts. This position will need to provide education on the

various components of the logic model as well to ensure that program managers are familiar with

the intent of each component. Also, the information technology resources of BFR will need to

support the implementation for these efforts and must be available to ensure that data tracking

and collection is not too cumbersome for individual programs to carry out.

Finally, a key finding of the research was that the regular reporting to external and

internal stakeholders, such as elected officials, was supported by the research and should be

considered in the establishment of an outcomes reporting program. The literature supported that

such integration aided in the accountability of these measures. Also, the respondents in the

survey instrument rated this as being directly related to their successful implementation of such

measures.

This paper was limited by the lack of participation of public sector organizations outside

of the fire service. Additional research should be aimed at identifying best practices outside the

fire service to ensure that the implementation of these measures is informed by a wider


experience in the public sector. In addition, future research should explore the statistic validity of

these measures to ensure that outcomes are linked with causation of program activities.


References

Baxter, S., Blank, L., Buckley-Woods, H., Payne, N., & Rimmer, M. (2014, May 10). Using

logic model methods in systematic review synthesis: describing complex pathways in

referral management interventions. BMC Medical Research Methodology, 14-62.

https://doi.org/doi.org/10.1186/1471-2288-14-62

Boulder Economic Council. (2018). Demographic and economic data: A snapshot of Boulder’s

key metrics. Retrieved April 2, 2018, from http://bouldereconomiccouncil.org/boulder-

economy/demographic-economic-data/

Boulder Economic Council. (n.d.). http://bouldereconomiccouncil.org/boulder-

economy/innovation-economy/

Boulder Fire-Rescue. (2018). Community risk assessment and standards of cover [Standards of

Cover]. Boulder, CO: Boulder Fire-Rescue Department.

Bruegman, R., & Lindsey, J. (2009). Fire administration. New York, NY: Pearson.

Budgeting for outcomes: Success in Baltimore. (2018). Retrieved from

https://www.innovations.harvard.edu/opex/research/budgeting-outcomes-success-

baltimore

City of Boulder Council website. (n.d.). https://bouldercolorado.gov/city-council/mayor-and-

city-council

Center for Public Safety Excellence. (2015). Fire and emergency services self-assessment

manual (9th ed.). Chantilly, VA: Author.

De Lancer Jules, P., & Holzer, M. (2002, December 17). Promoting the Utilization of

Performance Measures in Public Organizations: An Empirical Study of Factors Affecting


Adoption and Implementation. Public Administration Review, 61.

https://doi.org/https://doi.org/10.1111/0033-3352.00140

Earl, S., & Carden, F. (2002, August 1). Learning from complexity: The International

Development Research Centre`s experience with outcome mapping, https://

doi.org/10.1080/0961450220149852

Earl, S., Carden, F., & Smutylo, T. (2004). Outcome mapping: building and reflection into

development programs. Ottawa, ON: International Development Research Centre.

Engelke, P., & Paul, A. (2016). Boulder, Colorado: Innovation in a small town and a big state.

Retrieved from http://www.atlanticcouncil.org/blogs/futuresource/boulder-colorado-

innovation-in-a-small-town-and-a-big-state

Han, Y., & Hong, S. (2016). The impact of accountability on organizational performance in the

US Federal Government: The moderating role of autonomy. Review of Public

Administration, 1-21. https://doi.org/10.1177/0734371X16682816

Heinrich, C. (2002). Outcomes-based performance management in the public sector:

Implications for government accountability and effectiveness. Public Administration

Review, 62, 712-725. https://doi.org/doi/epdf/10.1111/1540-6210.00253

Innovation. (2018). Retrieved October 28, 2018, from https://bouldercolorado.gov/innovate

Iversen, J., & Ngwenjama, O. (2006). Problems in measuring effectiveness in software process

improvement: A longitudinal study of organizational change at Danske Data.

International Journal of Information Management, 26(1), 30-43.

https://doi.org/doi.org/10.1016/j.ijinfomgt.2005.10.006Get

https://doi.org/10.1080/0961450220149852


Joly, B., Polyak, G., Davis, M., & Brewster, J. (2007, August). Linking Accreditation and Public

Health Outcomes: A Logic Model Approach. Journal of Public Health Management, 13,

349-356. https://doi.org/Retrieved from

Kotter, J. P. (2012). Leading change. Boston, MA: Harvard Business Review Press.

Lee, R. D., Johnson, R. W., & Joyce, P. G. (2004). Public budgeting systems (7th ed.). Sudbury,

MA: Jones and Bartlett.

McLaughlin, J., & Jordan, G. (1999). Logic models: a tool for telling your programs

performance story. Evaluation and Program Planning, 22(1), 65-72. https://doi.org/

Milakovich, M. E., & Gordon, G. J. (2004). Public administration in America (8th ed.). Belmont,

CA: Wadsworth/Thomson Learning.

Newcomer, K. E. (2007, January 17). Measuring government performance. International Journal

of Public Administration, 30, 307-329.

https://doi.org//doi.org/10.1080/01900690601117804

Parsons, J., Gokey, C., & Thornton, M. (2013). Indicators of inputs, activities, outputs,

outcomes, and impacts in security and justice programming. Retrieved from Department

for International Development: https://assets.publishing.service.gov.uk/

government/uploads/system/uploads/attachment_data/file/304626/Indicators.pdf

Pearson, S. (2005). Outcomes measures implementation best practice guidance. Retrieved from

National Institute for Mental Health in England: https://www.researchgate.net/profile/

Peter_Fonagy/publication/272824369_Outcomes_Measures_Implementation_

Best_Practice_Guidance/links/54f5dae80cf2ca5efefd3aaf/Outcomes-Measures-

Implementation-Best-Practice-Guidance.pdf


Phillips, J. (2011). Return on investment in performance and improvement training programs.

(2nd ed.). New York, NY: Routledge.

Sheriff, N., & Schuetz, T. (2010). Benefits and challenges of applying outcome mapping in an

R4D project. Retrieved from World Fish Center: https://www.outcomemapping.ca/

download/WF_2625.pdf

Spekle, R. F., & Verbeeten, F. H. (2014, June). The use of performance measurement systems in

the public sector: Effects on performance. Management Accounting Research, 25(2),

131-146. https://doi.org/doi.org/10.1016/j.mar.2013.07.004

Taylor-Powell, E., & Henert, E. (2008). Developing a logic model: Teaching and training guide.

Retrieved, from https://peerta.acf.hhs.gov/sites/default/files/public/

uploaded_files/Logic%20Model%20Guide.pdf

Thomas, P. G. (2006). Performance measurement, reporting, obstacles and accountability:

Recent trends and future directions. Canberra, Australia: Australian National University

Press.

United States Fire Administration. (2015). Executive leadership student manual. Emmitsburg,

MD: United States Fire Administration.

University of Kansas. (2018). Community toolbox: Developing a logic model or theory of

change. Retrieved from https://ctb.ku.edu/en/table-of-contents/overview/models-for-

community-health-and-development/logic-model-development/main


Appendix A














Documents

Incorporating outcome measures into fire department