Collecting Costs of CommunityPrevention Programs

Communities Putting Prevention to Work InitiativeOlga A. Khavjou, MA, Amanda A. Honeycutt, PhD, Thomas J. Hoerger, PhD,

Justin G. Trogdon, PhD, Amanda J. Cash, DrPH

From the RTResearch TrianSecretary for P

Address co3040 Cornwaokhavjou@rti.o

0749-3797/http://dx.do

& 2014 Ame

Background: Community-based programs require substantial investments of resources; however,evaluations of these programs usually lack analyses of program costs. Costs of community-basedprograms reported in previous literature are limited and have been estimated retrospectively.

Purpose: To describe a prospective cost data collection approach developed for the CommunitiesPutting Prevention to Work (CPPW) program capturing costs for community-based tobacco useand obesity prevention strategies.

Methods: A web-based cost data collection instrument was developed using an activity-basedcosting approach. Respondents reported quarterly expenditures on labor; consultants; materials,travel, and services; overhead; partner efforts; and in-kind contributions. Costs were allocated acrossCPPW objectives and strategies organized around five categories: media, access, point of decision/promotion, price, and social support and services. The instrument was developed in 2010, quarterlydata collections took place in 2011–2013, and preliminary analysis was conducted in 2013.

Results: Preliminary descriptive statistics are presented for the cost data collected from 51respondents. More than 50% of program costs were for partner organizations, and over 20% of costswere for labor hours. Tobacco communities devoted the majority of their efforts to media strategies.Obesity communities spent more than half of their resources on access strategies.

Conclusions: Collecting accurate cost information on health promotion and disease preventionprograms presents many challenges. The approach presented in this paper is one of the first effortssuccessfully collecting these types of data and can be replicated for collecting costs from otherprograms.(Am J Prev Med 2014;](]):]]]–]]]) & 2014 American Journal of Preventive Medicine. All rights reserved.

Introduction

Disease prevention and health promotion havebeen increasingly emphasized as two of the mostimportant public health goals.1,2 Community-

or population-level prevention programs frequentlyimplement evidence-based strategies aimed at improvingthe public’s health. Given the varying levels of evidencerelated to these strategies, rigorous evaluations arenecessary to ensure that programs are effective and

I International (Khavjou, Honeycutt, Hoerger, Trogdon),gle Park, North Carolina; and the Office of the Assistantlanning and Evaluation (Cash), USDHHS, Washington DCrrespondence to: Olga A. Khavjou, MA, RTI International,llis Road, Research Triangle Park NC 27709. E-mail:rg.$36.00i.org/10.1016/j.amepre.2014.02.014

rican Journal of Preventive Medicine. All rights reserved.

achieve their goals. Although community and populationhealth programs frequently measure health-related out-comes, the costs of implementing large-scale preventionprograms have not been systematically assessed. TheUSDHHS believes that analysis of costs required toimplement these types of programs is an importantcomponent of a more comprehensive program evalua-tion and therefore undertook a study to examine costs ofstrategies implemented by the Communities PuttingPrevention to Work (CPPW) initiative. This informationwill assist in answering questions regarding the costs ofimplementing various types of prevention strategies indifferent communities.Community-based prevention programs that focus on

implementing population-wide strategies, such as nutri-tion signage in restaurants or access to tobacco-freeenvironments, are sometimes viewed as having no direct

Am J Prev Med 2014;](]):]]]–]]] 1

Khavjou et al / Am J Prev Med 2014;](]):]]]–]]]2

costs because public health delivery strategies do notincur direct medical costs for individual program partic-ipants.3 However, these types of programs require sub-stantial investments of resources. The publishedliterature on evaluations of community-based programs,specifically on program cost analyses, is limited.3 Pub-lished economic analyses of community-based programshave estimated costs retrospectively through interviewsof study authors and cost imputation for resourcesused.4,5

Subramanian et al.6 described a standardized instru-ment for cost data collection for a cancer screeningprogram; however, clinical interventions are inherentlydifferent from community-based programs. Specifically,community-based programs may require significantinfrastructure investments, often from multiple partners.For example, building a bike path includes constructioncosts but also requires involvement of organizations suchas transportation agencies, elected officials and boards,law officials, advocacy groups, or other local groups.This paper outlines a prospective cost data collection

approach and associated instrument that were developedfor the CPPW program. The study is one of the firstefforts to prospectively collect cost data for a variety ofinterventions aimed at improving population healthacross a large number of diverse communities. Thisapproach can be applied in other health promotion anddisease prevention programs, as program evaluationsthat include cost assessments are becoming increasinglyimportant.7

The CPPW ProgramIn 2010, the CDC funded 44 communities and states toparticipate in a CPPW initiative under the 2009 Amer-ican Recovery and Reinvestment Act (ARRA) (cdc.gov/CommunitiesPuttingPreventiontoWork/program/index.htm). This program was designed to make policy, system,and environmental changes to prevent or delay chronic

Table 1. Descriptions of the MAPPS categories

MAPPS category

Media Use media to promote healthy foods/drinkadvertising for tobacco and unhealthy food

Access Increase access to healthy food/drink choienvironment; reduce the availability of tob

Point of decision/promotion

Use point of decision labeling/signage/placonsumption of healthy foods/drinks, and

Price Use price to discourage consumption of to

Social support/services

Use social supports/services to promote to

MAPPS, media, access, point of decision/promotion, price, and social supp

disease and promote wellness by addressing the twoleading preventable causes of death and disability:tobacco use and obesity.8

Fourteen CPPW communities received awards todecrease smoking prevalence, teen smoking initiation,and exposure to secondhand smoke; twenty-three com-munities received awards to increase levels of physicalactivity, improve nutrition, and decrease obesity rates;and seven communities received awards for both tobaccoand obesity prevention efforts. Communities wereinstructed to implement evidence-based strategies in fivecategories, collectively referred to as media, access, pointof decision/promotion, price, and social support/services(MAPPS) (http://www.cdc.gov/chronicdisease/recovery/PDF/MAPPS_Intervention_Table.pdf). MAPPS categorydescriptions are shown in Table 1. Communities wererequired to comply with quarterly ARRA reportingrequirements (recovery.gov/Pages/default.aspx). Addi-tional communities were later funded under the Afford-able Care Act (ACA); however, the study described hereis based on data from the 44 originally funded ARRAcommunities. Additional information about the CPPWprogram has been published elsewhere.A rigorous evaluation of CPPW was crucial to ensure

that strategies used in the funded communities arereproducible and sustainable in other communities.The cost study described in this paper is one componentof a larger, comprehensive effort to evaluate this pro-gram. The cost study was funded by the USDHHS’sOffice of the Assistant Secretary for Planning andEvaluation and the CDC and contracted to RTIInternational.

MethodsThe CPPW cost data collection effort used an activity-basedcosting approach.6,9–11 This approach enables cost estimation foreach main program activity and requires data collection on all

Description

s and increase activity; restrict advertising and employ counter-s/drinks.

ces and safe locations to be active and improve the builtacco and unhealthy foods/drinks.

cement to discourage consumption of tobacco, increaseprompt physical activity.

bacco and to benefit consumption of healthy foods/drinks.

bacco cessation, breastfeeding, and increased activity.

ort/services

www.ajpmonline.org

Khavjou et al / Am J Prev Med 2014;](]):]]]–]]] 3

resources used for the program by activity. With this approach,program activities must be defined in a way that is meaningful torespondents (i.e., program work is organized around theseactivities) and represents categories that are common and can bestandardized across multiple respondents. A challenge in design-ing the CPPW cost data collection approach was identifyingprogram activities that satisfied both of these requirements,particularly given the variability in community strategies.CPPW communities developed plans that described the specific

objectives of their CPPW programs.8 However, each communitydeveloped its own CPPW objectives rather than selecting themfrom a uniform list, limiting the overlap in objectives acrosscommunities. Moreover, the number of objectives varied acrosscommunities from six to 30. Organizing cost data collectionaround objectives was meaningful for communities; however, itcreated an analytic challenge because objectives were not commonacross communities and thus the objective costs could not becompared easily.This issue was addressed by requesting additional information

about objective costs and whether they supported specifiedMAPPS strategies. Communities were initially instructed to workfrom a prescribed menu of 45 MAPPS strategies, which provideda common element for cost data collection across communities.Communities were required to report costs at the objective levelbut also to specify which MAPPS strategies under each objectivethe costs supported.A web-based cost data collection tool, the Cost Study Instru-

ment (CSI), was developed to collect cost data from CPPWcommunities quarterly. The CSI was developed with the ColdFu-sion rapid application development platform (Adobe ColdFusion8, Adobe Systems Incorporated, San Jose CA) in concert witha Structured Query Language (SQL) Server database (MicrosoftSQL Server v. 9.00.5324.00, Microsoft, Redmond WA). Theinstrument has administrative and authentication modules, allow-ing for individual respondent log-in and account management.The SQL Server database allowed for flexible and rapid datareporting for analysis. The CSI was developed in 2010, quarterlydata collections took place between 2011 and 2013, and dataanalysis was conducted in 2013.A draft version of the CSI was pretested with seven CPPW

respondents. During the pretest, the clarity of the instrument,usability of the system, and accuracy of the entered data wereevaluated. Pretesters were asked to complete individual CSIscreens and provide feedback on the content and functionality ofthe tool by completing a questionnaire and participating in aconference call. The CSI development team also worked with thepretesters on reviewing examples of bills and invoices and howthey should be entered in the CSI. All communities receivedtraining on cost data collection and reporting and the use of theCSI during a 1-day site visit. A user’s guide was developed,providing variable definitions and comprehensive instructions fordata entry and submission.The CSI collected quarterly expenditures for the following

resource components: (1) labor/personnel; (2) consultants; (3)materials, travel, and services; (4) indirect overhead; and (5)partner organization efforts (i.e., contracted services). Thesecomponents have been used in other cost collection studies,6,9,12

as they parallel common budget categories familiar to manage-ment and fiscal staff. Labor and non-labor in-kind contributions ofgroups and individuals that partnered with the communities were

] 2014

also captured in the CSI, where “in-kind” represents efforts thatsupported CPPW program activities but were not paid for by theCPPW grant. It was important to track the value of these “free”resources in addition to actual program outlays to ensure that theestimates captured the full economic cost of CPPW programefforts.In the CSI, respondents were required to allocate costs for each

resource component except overhead costs across a community’sCPPW objectives as defined in community work plans. Respond-ents entered percentage allocations of each payment across theobjectives. For example, if $10,000 was paid to a partner who spent90% of their time and efforts on Objective 1 and 10% on Objective2, the respondent entered $10,000 in the quarterly payment celland 90% and 10% as allocations for Objectives 1 and 2,respectively.Within each objective, respondents also used checkmarks to

indicate which MAPPS strategies linked to the objective weresupported by the objective-level costs. If an objective was tied toonly one strategy, then 100% of the objective-level cost wasassigned to that strategy. When more than one strategy was usedto achieve an objective, equal allocations of costs were assumedacross the strategies. For example, if an objective was tied to threestrategies but only two of them were checked for a specific costentry (e.g., quarterly payment to a partner), then the objective-levelcost was split equally across the two checked strategies. SelectedCSI screen shots are presented in Appendix A.Additional cost allocation categories included “evaluation” and

“administrative.” The CDC required that communities evaluateefforts; the CSI captured these costs separately because these werenonprogrammatic costs not required for program implementation.Administrative costs were those associated with administrationand management of the grant and other activities not devoted tospecific CPPW objectives or evaluation.When designing the data collection system, we aimed to

maximize the clarity and comprehensibility of the instrument,minimize the possibility of data entry errors, and minimize theburden on respondents of completing the data entry. The first aimwas addressed with in-person trainings, a user’s guide, andongoing technical support via e-mail, phone calls, and webinarsprovided to communities. Also, the instrument was developedwith an intuitive and menu-driven design.To address the second aim, the CSI included automated checks

to avoid minor user errors and ensure that data entries were withinthe anticipated ranges. These features allowed respondents tocorrect data entry mistakes prior to submission. Examples of theseautomated checks included confirming that the percentage allo-cations of costs across objectives added up to 100% and that at leastone strategy under an objective was checked if an objective wasassigned a non-zero percentage allocation.To minimize respondent burden, the CSI was designed to use

data that respondents had already provided to other parts of theCPPW evaluation so that they would not need to re-enter the samedata into the CSI. For example, data on each community’sobjectives and strategies were preloaded into the CSI from adatabase managed by the CDC. Partner names from communities’budgets were also preloaded in the CSI. Additionally, the CSIallowed respondents to pre-load entries, such as job titles,consultant names, and percentage cost allocations across objec-tives, from a previous data submission. Finally, the CSI wasdynamic; the most recent versions of community objectives were

Partner, 55%

Labor, 21%

Materials, travel, and services, 13%

Overhead/indirect, 7%

In-kind, 4%Consultants, 1%

Figure 1. Total costs by resource category, tobaccocommunities

Khavjou et al / Am J Prev Med 2014;](]):]]]–]]]4

loaded into the CSI every quarter to capture changes in commun-ity efforts. The average burden to complete the study was estimatedat 13 hours per year per respondent.

ResultsResults include preliminary descriptive statistics for costsincurred from March 18, 2010, through March 31, 2012.Analysis of the data from the communities that receivedan extension of the grant without additional funds for thecosts incurred after March 31, 2012, is in progress.Respondents included the 44 CPPW communities

originally funded under ARRA. Each of the seven

Partner, 52%

Labor, 21%

Materials, travel, and services, 9%

In-kind, 8%

Overhead/ indirect, 6%

Consultants, 4%

Figure 2. Total costs by resource category, obesitycommunities

communities that received funding for both tobaccoand obesity were treated as two separate respondents,one for each grant, thus bringing the total number of coststudy respondents to 51.Figures 1 and 2 depict CPPW costs by resource

category among tobacco and obesity communities,respectively. More than half of the CPPW program costswere paid to partner organizations: 55% in tobacco and52% in obesity communities. Slightly more than 20% ofcosts were for CPPW labor hours. In-kind contributionscomprised a higher percentage of total costs in obesitycommunities (8%) than tobacco communities (4%). Theshare of costs for materials, travel, and services washigher in tobacco communities than obesity commun-ities (13% vs 9%), which likely reflects implementation ofsocial support and services strategies through the pur-chase of supplies for quitline services and nicotine-replacement therapy by tobacco communities.Figures 3 and 4 show total spending by MAPPS

category among tobacco and obesity communities. Onaverage, tobacco communities devoted the majority oftheir efforts to media (40%), with 26% of costs for socialsupport and services and 26% for limiting access totobacco products. In contrast, obesity communities spentmore than half of their resources on improving access tobetter nutrition and physical activity (51%–55%) andonly about one quarter on media (22%–26%).For physical activity efforts, the proportion of costs for

social support and services was double that for nutritionefforts (16% vs 8%). This difference is likely driven by thefact that breastfeeding was the only strategy under thenutrition social support and services category whereas forphysical activity, this category included activity groups

Media, 40%

Access, 26%

Point of decision/ promo�on, 4%

Price, 4%

Social support and services, 26%

Figure 3. Total costs by MAPPS category, tobacco commu-nitiesMAPPS, media, access, point of decision/promotion, price, and socialsupport/services

www.ajpmonline.org

a. Physical Activity

b. Nutrition

Media, 22%

Access, 55%

Point of decision/

promo�on, 4%

Price, 3%Social support and services,

16%

Media, 26%

Access, 51%

Point of decision/

promo�on, 8%

Price, 7%

Social support and services,

8%

Figure 4. Total costs by MAPPS category, obesity commu-nities: physical activity and nutrition initiativesMAPPS, media, access, point of decision/promotion, price, and socialsupport/services

Khavjou et al / Am J Prev Med 2014;](]):]]]–]]] 5

and safe routes to school strategies. The proportion ofcosts spent on pricing strategies in physical activityefforts was less than half of that for pricing strategies innutrition efforts (3% vs 7%).

DiscussionAweb-based data collection instrument was developed tocollect and report quarterly costs incurred by the CPPWcommunities working on tobacco use and obesity pre-vention efforts. The biggest challenge was to collect

] 2014

accurate and uniform data from 51 respondents with alarge variability between the strategies undertaken thatcould be aggregated and standardized to address theresearch questions for the study.A number of advantages and limitations associated

with the use of a web-based cost data collection instru-ment were identified. Advantages include having acentral data repository, which simplifies aggregation ofdata across respondents and over time, system checks forminor data entry errors, and system flexibility. In the caseof the CPPW cost data collection, the web-based featureswere especially useful because they allowed accommo-dation of a wide range of strategies undertaken by CPPWcommunities and modification of the instrument asneeded to reflect ongoing changes in community efforts.Another advantage for the users was the ability to preloaddata from other data sources and previous submissions.A major limitation in collecting cost data from the

CPPW communities was the limited CPPW communitystaff expertise and time available to collect and reportprogram costs. Few community programs had previouslyreported these data, especially not with the frequencymandated by ARRA funding. To address this limitation,the CSI was designed to include features that eased dataentry and minimized respondent burden, at the sametime providing for collection of accurate and completecost data.In-person training was also provided at the beginning

of the project as well as substantial technical assistanceduring each cycle of data submission. Once the cost datawere submitted, the technical assistance team conductedadditional data quality reviews. These examinationsfocused on implausible data and any potential incon-sistencies or inaccuracies. Reviewers noted any issues andfollowed up with respondents to confirm or correct dataentries.Another limitation is that collecting this level of cost

data requires significant resources, especially program-ming, to develop, de-bug, and pretest an instrumentbefore it is ready for end users. Many preventioninitiatives require evaluations; however, the cost studywas one of the first of its kind and required significantcosts to build the CSI and provide ongoing technicalassistance. Forethought should be given to any programbefore it is implemented if cost data are of interest.An additional limitation is that the accuracy of cost

allocations to specific objectives and thus the validity ofthe CSI cannot be verified. Although this is a commonlimitation of cost data collection instruments,6 the levelof technical assistance provided to the respondents in thisstudy and reviews of data submission quality should havefacilitated accurate reporting (see Appendix B for moredetails). Furthermore, because we faced a trade-off

Khavjou et al / Am J Prev Med 2014;](]):]]]–]]]6

between respondent burden and data complexity, weassumed equal allocations if multiple strategies werechecked under the same objectives, instead of askingfor specific percentage allocations for each category.A number of important lessons were learned as data

were collected from the CPPW communities. First,ongoing technical assistance was essential to ensurecompliance with reporting requests and allow for datasubmission quality checks. Second, detailed testing of theCSI was needed before making the instrument availableto communities for web-based reporting. Third, the datacollection system should be flexible enough to accommo-date ongoing changes in community efforts, becauseactivities, strategies, and interventions can change over timeowing to improvements or knowledge gained. Frequentmonitoring and updates were needed to collect accuratecosts while maintaining changes in community efforts.Fourth, although training and setup of cost reporting

infrastructure can be time consuming, communities wereable to report the required level of data without signifi-cant burden. Finally, in-person training site visits wereimportant in establishing rapport and strong workingrelationships with communities, which improved com-pliance with the study requirements.Collecting accurate and comprehensive data on the

cost of the prevention strategies CPPW communitiesimplemented was a critical first step in assessing costs forthe wide variety of CPPW strategies. After all datacollection is complete, the collected data will answerquestions that may identify community and programfactors related to costs, how costs are affected whenvarious combinations of related strategies are imple-mented in a community, and whether costs for thesetypes of strategies decrease as more people are reached.As federal programs aim to fund and implement

evidence-based strategies that are cost-effective, carefulevaluations of these types of programs are also increas-ingly important. The IOM Committee on Public HealthStrategies to Improve Health recommends developmentof data systems and measures to capture information onkey elements of public health delivery, including costs ofprograms and strategies aimed to improve populationhealth.7

Although other instruments are available for collectingcosts of clinical interventions,6 the cost data collectionapproach and system that was developed for the CPPWprogram was a first step toward understanding howto build a system that collects accurate data on costsfor an initiative aimed at improving population healthwhile minimizing the burden placed on respondents tocollect and report the data. This approach can be appliedto other health promotion and disease prevention

programs, as cost evaluations should be integrated intocomprehensive program evaluations. Future work shouldexplore the feasibility of validating the data collectioninstrument presented here.

This report was prepared by RTI, under contract to theAssistant Secretary for Planning and Evaluation (ASPE). Thefindings and conclusions of this report are those of the authorsand do not necessarily represent the views of the ASPE, CDC,or USDHHS.No financial disclosures were reported by the authors of

this paper.

References1. National Prevention Council. National prevention strategy. Washing-

ton DC: USDHHS, Office of the Surgeon General, 2011.2. Merzel C, D’Afflitti J. Reconsidering community-based health promo-

tion: promise, performance, and potential. Am J Public Health 2003;93(4):557–74.

3. Trust for America’s Health (TFAH). Prevention for a healthierAmerica: investments in disease prevention yield significant savings,stronger communities. Washington DC: Trust for America’s Health,2009.

4. Roux L, Pratt M, Tengs TO, et al. Cost effectiveness of community-based physical activity interventions. Am J Prev Med 2008;35(6):578–88.

5. Wu S, Cohen D, Shi Y, Pearson M, Sturm R. Economic analysis ofphysical activity interventions. Am J Prev Med 2011;40(2):149–58.

6. Subramanian S, Ekwueme DU, Gardner JG, Trogdon J. Developingand testing a cost-assessment tool for cancer screening programs. Am JPrev Med 2009;37(3):242–7.

7. IOM. For the public’s health: investing in healthier future. WashingtonDC: National Academies Press, 2012.

8. Bunnell R, O’Neil D, Soler R, et al. Fifty communities puttingprevention to work: accelerating chronic disease prevention throughpolicy, systems and environmental change. J Community Health2012;37(5):1081–90.

9. Finkelstein EA, Khavjou O, Will JC. Cost-effectiveness of WISE-WOMAN, a program aimed at reducing heart disease risk among low-income women. J Womens Health 2006;15(4):379–89.

10. Cho BH, Hicks KA, Honeycutt AA, et al. A tool for the economicanalysis of mass prophylaxis operations with an application to H1N1influenza vaccination clinics. J Public Health Manag Pract 2011;17(1):E22–E28.

11. Honeycutt AA, Clayton L, Khavjou O, et al. Guide to analyzing thecost-effectiveness of community public health prevention approaches.Research Triangle Park NC: RTI International, 2006.

12. Zarkin G, Dunlap J, Homsi G. The substance abuse services costanalysis program (SASCAP): a new method for estimating drugtreatment services costs. Evaluation and Program Planning 2004;27(1):35–43.

Appendix

Supplementary data

Supplementary data associated with this article can be found athttp://dx.doi.org/10.1016/j.amepre.2014.02.014.

www.ajpmonline.org