b09c074129f1943b4a172d23fb8542b5

8/9/2019 b09c074129f1943b4a172d23fb8542b5

1/24

e x c h a n g e

Several years ago Ivisited India andlearned about GyanShala, a nonpro t start-up that developed anentrepreneurial low-costmodel or providing high-quality education to poorchildren. It was ascinat-ing to hear the story o how this model scaled up

over time, starting rst with local implemen-tation at the elementary grades, moving nextto middle school grades, and nally spreadingnationally, as Gyan Shala worked with the gov-ernment to integrate the model into the publiceducation system. During my career, I have seennumerous examples o programs that were suc-cess ul as research and demonstration e orts,but then died when they attempted to scale up.The experience o Gyan Shala made me wantto take another look at what we know aboutsuccess ully taking an e ort to scale.

Scaling up involves sharing somethingthat is e ective (such as programs, practices,or ideas) so that more people can experienceits bene ts. When something works well, thenatural inclination is to share it. The challengebecomes how to do so e ectivelyand that iswhere evaluation comes in.

This issue o The Evaluation Exchangeexplores the promising practices and challengesassociated with taking an enterprise to scale,along with the role that evaluation can andshould play in that process. It is the second inour hard-to-measure series, which we inau-gurated with our Spring 2007 issue on evaluat-ing advocacy.

Surprisingly ew examples exist o nonpro te orts that have scaled up and achieved lastingsuccess. A program or approach may be strongand e ective in one location, but that does notmean it will work the same way in another.Scaling is a complex process that plays outwithout a script. But we do know that when we

e v a l u a t i o n

h d i r e c t o r s d e s k

h

h s ssScaling Impact

The & P a t e

Broadening the perspective 2

Six steps to scale impact 4

Be n Bas T a n ng

Five meanings o scale 7

SCALERS model 8

E a at ns t Wat h

Save the Children 9

Ask the Expe t

Lessons about scale 10

Q est ns & Answe s

A conversation with 12Mike Smith

P s ng P a t es

Evaluation to support scale 14

Integrated school evaluation 15

WINGS per ormance data 16

A broader concept o scale 17

RALLY scaling approach 18

Sp t ght

O ce o Social Innovation 19

eNonpro ts study 20

Why Facebook matters 21

New & N tew th 23

En N tes

Scaling impact 24

Heather Weiss

Heather B. Weiss, Ed.D.Founder & DirectorHarvard Family Research Project

take something to scale, we need to start witha clear sense o what is being scaled, why itbeing scaled, how the process will work, anwhat it should look like in the end. This issue oThe Evaluation Exchange helps readers thi

through some o those questions and optionsSeveral articles in the issue make it cle

that evaluation is integral to the scaling process. At the beginning o a scaling e

or example, evaluation can help determinwhether something is ready to go to scale anwhich o its components should be scaleThis stage includes identi ying and assessithe ingredients that must be in place to suc-cess ully scale.

In addition, this issue discusses how tevaluate an e ort during the scaling proceScaling takes place in developmental staes, and in ormation needs di er over timEvaluation questions and methods must recognize and assess those developmental stageaccordingly.

Other articles consider the lessons that experienced programs and evaluators have learnedabout the scaling process. These lessons drawon successes and ailures alike and are o eto help others navigate the process.

Finally, the issue addresses how to evaluate the di erent approaches that nonprouse to take something to scale. Several optionexist, each with di erent implications evaluation.

We hope that this issue o The EvaluatiExchange contributes to the conversatioabout scale in the nonpro t sector and, in par-ticular, reminds us o the important role thaevaluation can play in its success.

HArvArd FAmily rESEArcH ProjEcT HArvArd GrAduATE ScHool oF EducATioN volumE Xv NumBEr 1 SPriNG 2010

A P e r i o d i c A l o n e m e r g i n g S t r A t e g i e S i n e v A l u A t i o n

8/9/2019 b09c074129f1943b4a172d23fb8542b5

2/24

8/9/2019 b09c074129f1943b4a172d23fb8542b5

3/24

Harvard Family Research Project The Evaluation Exchange XV 13

Selected Resources on Theory

&> t h e o r y p r a c t i c eFour APProAcHES To ScAlE

What is S a e de n t n S a e S a ngme han s s

re e ant The /l te at e

inf en ng va ab es

P g a

A sys p j s s s ha s a

a s s.

c py a p a ha s a h hassh w b , w h h xp a-

ha a w p h s as s p a s. S a p a s

a w x b y p a b s a ap h a x .

r p aA ap a

P a th yth y cha

F y i p ag a zab y

A ap ab y

i ea inn at n

A w way h kab s h ; ws s p b s.

Sp a a a a a s a za s w h a a a a

sys ( aph , a za a ,p ss a ); as a b a ap f

p p s s x s.

c a

ma k

d ss a

d s i a

t pp P

c a s

th y

r a A a a

c pa b y

c p x y

t a ab y

obs ab y

Te hn g Sk

P s, s,h q s, p a s.

i as h b p p p a sha s app y a h y, p a ,

app a h.

ma k

d s b

t a

g a

K w t a s

t h y t a s

l a th y

r s s

Ab y

ma k d a

ex s

S pp PA a ab y

t h a S pp

P

c f s a s haf p a s a s a .

e s ha as xp ss as p y aa s b ha h h a

p a j s ( . ., y, y,s a , , y).

i p a A a S

P y i p a

th y

S p a Fa

g a

e

A ab y

r s s

A ap o s , J. m., & Haw , P. (2009). K w za , s , p a , a s , a a s a : i p a s a a .New Directions for Evaluation, 124, 1108.

ProgramFulbright-Anderson, K., Kubisch, A., & Connell, J. (Eds.).(1998). New approaches to evaluating community initiatives:Theory, measurement, and analysis . Washington, DC: TheAspen Institute.

Rogers, P. J., Hacsi, T. A., Petrosino, A., & Huebner, T. A.

(Eds.). (2000). Program theory in evaluation: Challenges andopportunities. New Directions in Evaluation, 87, 1112.

Idea or InnovationGladwell, M. (2002). The tipping point: How little things canmake a big di erence . New York: Back Bay Books.

Rogers, E. (2003). Di usion o innovations (5th ed.) . NewYork: Free Press.

Technology or Skill Bozeman, B. (1988). Evaluating technology trans er and di u-sion. Evaluation and Program Planning, 11 (1), 63104.

Goh, S. C. (2002). Managing e ective knowledge trans er: Anintegrative ramework and some practice implications. Journal o Knowledge Management, 6 (1), 2330.

PolicyGoggin, M. L., Bowman, A. O., Lester, J. P., & OToole, L. J.(1990). Implementation theory and practice: Toward a third

generation. Glenview, IL: Scott Foresman.

Sabatier , P. A. (2007). Theories o the policy process (2nd ed.).Boulder: Westview Press.

8/9/2019 b09c074129f1943b4a172d23fb8542b5

4/24


&> t h e o r y p r a c t i c e

Rather than starting

rom scratch, it o ten

makes more sense to take

a model that works well

in one community and try

it out in another.

Six Steps to Success ully Scale Impactin the Nonproft Sector

Erin Harris o Harvard Family Research Project discusses hownonpro ts can success ully scale up an intervention, based on areview o the literature on this topic.

T o increase their impact, many nonpro ts seek to scaleup or go to scale by expanding their interventions toreach larger populations. This scaling process most com-monly involves implementing the intervention at new sites orexpanding the capacity o existing sites to serve a larger numbero participants.

So why go to scale? For nonpro ts, the primary goal o scalingis to spread the impact o the intervention. Scaling can also be apractical approach: Rather than starting rom scratch, it o tenmakes more sense to take a model that works well in one com-munity and try it out in another. Interventions that go to scalecan also bene t rom being part o a larger network where theycan share resources and operating procedures.This larger network can o ten allow an inter-vention to produce bigger outcomes and at a

aster pace than each individual site would beable to do on its own. Further, demonstratingimpact on a larger scale can help create greatervisibility and leverage in attracting additionalsupport or the intervention.

Despite these bene ts, many interventionsail to success ully go to scale. The process o

scaling is not easy; it requires stakeholder sup-port and buy-in, care ul planning and assess-

ment, and su cient resources to maintainquality. This article describes the steps thatnonpro ts should ollow in going to scale to help ensure success.These recommended steps are based on a review o the litera-ture on scale (including the articles in this Evaluation Exchange issue).

1. Determine whether the intervention is ready to go to scale.There are several issues to consider in determining whether anintervention is ready or the scaling process. First and oremost,the intervention must have a strong theory o change or logicmodel that links the program inputs to the outcomes. This theoryo change or logic model should convey which components are

essential to the interventions success and which can allow orfexibility so that they can be tailored to local needs and availableresources.

Second, the nonpro t needs to test the theory o change orlogic model through evaluation o program implementation andoutcomes to determine: (a) whether the intervention was imple-mented as intended, (b) whether the intervention was e ective,(c) why the intervention was e ective (i.e., what elements o theintervention were linked to its success), (d) how the elements thatwere linked to success can be trans erred to new settings, and (e)

whether the interventions e ects were large enough to persist inthe ace o contextual variability.

Third, the intervention must have buy-in rom stakeholders(including unders) or the scaling process. It is crucial that theintervention have support, especially in the orm o additional

unding and other necessary resources to ensure that the qualityo the services and sta can be maintained at additional sites.Nonpro ts can use the results o the evaluation to help convince

unders and other key stakeholders o the value o the programand o how these bene ts can be spread to new sites.

2. Select the best approach to bring the intervention to scale.The term scale is used in the nonpro t sector to describe di er-ent approaches or expanding an interventions impact. Thoughthere is no one right way to scale, it is important or nonpro ts toconsider all o the possible approaches or going to scale and then

choose the one that best meets their needs.In the business world, bringing something

to scale usually re ers to replicating something(e.g., a business model or service) in a new sitewith as much delity to the original as possible.As applied to the nonpro t sector, however,replication tends to have a more nuanced mean-ing, where the goal is to replicate the programsimpact rather than its components exactly asthey were initially implemented. Replication isthe most straight orward way o conceptualiz-ing scale and allows nonpro ts to bene t rom

the lessons rom the business sector. However,replication as it applies to the nonpro t sec-

tor has been criticized or its narrow scope, in not viewing scalebeyond the goal o growth in numbers served.

Another approach that has grown in popularity in the nonpro -it sector is to scale ideas, regardless o whether those ideas includethe branding o the original intervention. This broader view o scale takes into account the act that ownership o the initiativeis less o a motivating actor in the nonpro t sector than in thebusiness world. It is o ten discussed using the term, the travelingo ideas, meaning that promising practices are communicated toothers in a given eld to help di use new concepts, processes,and techniques. 1 This approach allows or greater adaptability

to di erent settings and contexts.A third model o thinking about scale in the nonpro t sector isto view it as having multiple acets, with various ways o scalingan interventions impact. For example, Cynthia Coburn describesscale as including our components: spread, depth, sustainability,and shi t in re orm ownership (see the article, Applying a BroaderConcept o Scale to Evaluate a Funding Strategy, on page 17).

1. Scott, W. R., Deschenes, S., Hopkins, K., Newman, A., & McLaughlin, M.(2006). Advocacy organizations and the eld o youth services: Ongoing e orts torestructure a eld. Nonpro t and Voluntary Sector Quarterly, 35 (1), 691714.

8/9/2019 b09c074129f1943b4a172d23fb8542b5

5/24


&> t h e o r y p r a c t i c e

It is crucial that the

intervention have support,

especially in the orm o

additional unding,

to ensure that the quality

o the services and sta

can be maintained at

additional sites.

Similarly, Peter Frumkin describes scale as having ve di erentmeanings: nancial strength, program expansion, comprehensive-ness, multisite replication, and accepted doctrine (see the article,The Five Meanings o Scale in Philanthropy, on page 7). Both o these models help get at the complexities o scale in the nonpro tsector and create a conception o scale that is more meaning uland sustainable than simple replication.

3. Select sites that are best suited to the intervention.As indicated above, the interventions theory o change shouldprovide in ormation about what elements o the intervention arenecessary or its success and where there is room or negotiationand fexibility. For example, in selecting newsites to implement the Nurse-Family Partner-ship program, the program developer, DavidOlds, determined that the new sites had tohave certain capacities to operate and sus-tain the program with high quality. In par-ticular, the sites had to have the capacity oran organization and community that are

ully knowledgeable and supportive o theprogram; a sta that is well trained and sup-ported in the conduct o the program model;and real-time in ormation on implementationo the program and its achievement o bench-marks to guide e orts in continuous qual-ity improvement. 2 As part o the selectionprocess, nonpro ts should ensure that thecommunities being considered or the newsites have the local capacity to support theelements o the intervention that are necessary or its success.

In addition, the intervention should be implemented in commu-

nities where there is an identi ed need that is not being su cientlymet by existing community resources. Thus, the site-selection pro-cess should include ensuring that the program is not moving intoan area that is already saturated with similar services. Otherwise,the intervention is likely to nd itsel competing or participantsand local resources (e.g., quality sta ) while adding little value tothe local community.

4. Develop the capacity and in rastructure to managemultiple sites.Managing multiple sites requires additional structures and proce-dures to ensure that these programs are e ectively and e cientlymanaged across the larger system. Several pieces must be in placeto ensure that the organizational in rastructure can support thescaling process.

First, nonpro ts need to develop management structures tocoordinate services both across and within sites. In particular,

2. Olds, D. L. (2007). The Nurse-Family Partnership: From trials to practice , p. 27.Federal Research Paper. Online at: www.earlychildhoodrc.org/events/presentations/ olds.pd . For more in ormation on the scaling process o the Nurse-Family Partner-ship, see: Goodman, A. (2006). The story o David Olds and the Nurse Home Visit-ing Program . Princeton: The Robert Wood Johnson Foundation. Online at: www.rwj .org/ les/publications/other/DavidOldsSpecialReport0606.pd .

the intervention must develop and communicate roles or locasites and central o ces so that it is clear who is responsible owhich aspects o running the program. Systems must be in place

or communication across the various levels so that everyone iworking toward a common set o goals as part o a coordinatedteam e ort.

Second, resources and structures must be in place at the locallevel that can help ensure quality, such as strong local leadershipand quali ed service delivery sta . This piece o ten proves toone o the most challenging issues or nonpro ts in scaling: how tmaintain quality as the program grows. However, i the sites havebeen care ully selected as those best suited to the intervention

theory o change, quality control will be leo a problem. Still, program developers neeto be vigilant in ensuring that quality standards are met and upheld. To ensure qualitynonpro ts need to allocate resources to evaluate implementation and outcomes across site(see step 5 below or more detail).

Third, program leaders should be awarand take advantage o resources and strengthin the local community that can bene t thintervention. At the same time, programdevelopers should be aware o de cits in thlocal setting that they will need to overcom(or accommodate). To deal with the local context, fexibility in implementation can helpstrengthen the intervention. For examplean a ter school program being implementein a community with a strong school system

might consider nding ways to partner with local schools to hireschool-day teachers as program sta . That same program migh

also want to work with other local a ter school programs to coor-dinate their services to complement (rather than duplicate) oneanother. Strategies will di er rom site to site, depending on thstrengths and weaknesses o the local context.

5. Evaluate the scaling process.As the intervention is implemented at new sites, the scaling processitsel should be evaluated. Implementation evaluation conductedduring the scaling process should ocus on continuous learningand improvement; as part o this process, the evaluation ndingcan help in orm any necessary adjustments that need to be madeto the program to account or the local context. These adjust-ments are o ten inevitable; even with a strong theory o changeit can be di cult, i not impossible, to anticipate all the possiblways in which the local context may a ect program implementation and outcomes.

Evaluation can also take place at the end o the scaling procesto determine the successes (and ailures) o the process and whaled to those successes and ailures. Nonpro ts can then use theslessons to help in orm the interventions uture scaling strategieand to provide guidance to other interventions in their scalingprocesses. They can also use positive outcomes to promote theintervention to potential new stakeholders (including unders
http://www.earlychildhoodrc.org/events/presentations/olds.pdfhttp://www.earlychildhoodrc.org/events/presentations/olds.pdfhttp://www.earlychildhoodrc.org/events/presentations/olds.pdfhttp://www.rwjf.org/files/publications/other/DavidOldsSpecialReport0606.pdfhttp://www.rwjf.org/files/publications/other/DavidOldsSpecialReport0606.pdfhttp://www.rwjf.org/files/publications/other/DavidOldsSpecialReport0606.pdfhttp://www.rwjf.org/files/publications/other/DavidOldsSpecialReport0606.pdfhttp://www.earlychildhoodrc.org/events/presentations/olds.pdfhttp://www.earlychildhoodrc.org/events/presentations/olds.pdf

8/9/2019 b09c074129f1943b4a172d23fb8542b5

6/24


Bradach, J. (2003, Spring). Going to scale. The Stan ord Social Innovation Review , 1825 . Online at: www.ssi review.org/articles/entry/going_to_scale/ .

Campbell, K., Ta t-Pearman, M., & Lee, M. (2008). Getting replication right: The decisions that matter most or nonpro -it organizations looking to expand. New York: BridgespanGroup. Online at: www.bridgespan.org/LearningCenter/ ResourceDetail.aspx?id=398 .

Carlson, N. F. (2008). Replicating success: A unders per-spective on the why and how o supporting the local o ce o an expanding organization. A case study o BlueRidge Foundation New Yorks support o the Taproot Foun-dation NYC . Brooklyn: Blue Ridge Foundation New York.Online at: www.br ny.org/downloads/BlueRidge-Taproot FoundationCaseStudy.doc .

The Center or the Advancement o Social Entrepre-neurship. (2006). Scaling social impact research project thought leader meeting annotated bibliography, v. 1.0. Durham, NC: Duke University Fuqua School o Business.Online at: www.caseatduke.org/knowledge/scalingsocial impact/bibliography.html .

Clarke, J., Dede, C., Ketelhut, D. J., & Nelson, B. (2006).A design-based research strategy to promote scalability oreducational innovations. Educational Technology, 46 (3),2736. Online at: http://muve.gse.harvard.edu/rivercity project/documents/ nal_scale_article_Clarke.pd .

Coburn, C. (2003). Rethinking scale: Moving beyond num-bers to deep and lasting change. Educational Researcher,32 (6), 312. Online at: http://gse.berkeley.edu/ aculty/ CECoburn/coburnscale.pd .

Creech, H. (2008). SEED Research and Learning Pro- gramme 20078: Scale up and replication or social and environmental enterprises . Gland, Switzerland: The SEED

Initiative. Online at: www.seedinit.org/download-document/ 13-seed-iisd-report-on-scale-up-and-replication-280708sml .html .

Edna McConnell Clark Foundation. (2008 ). An experi-ment in coordinated investment: A progress report on theEdna McConnell Clark Foundations growth capital aggre-

gation pilot. New York: Author. Online at: www.emc .org/ pd /gcap_progressreportOct08.pd .

Elmore, R. E. (1996, Spring). Going to scale with goodeducational practice. Harvard Educational Review, 66 (1),127 .

Farole Jr., D. J. (2006.) The challenges o going to scale:Lessons rom other disciplines or problem-solving courts. New York: Center or Court Innovation. Online at: www.courtinnovation.org/_uploads/documents/Lessons.pd .McDonald, S . (2009) . Scale-up as a ramework or inter-vention, program, and policy evaluation research. InG. Sykes, B. Schneider, & D. N. Plank (Eds.), Handbook o education policy research (pp. 191208). New York: Rout-ledge Publishers ( or the American Educational ResearchAssociation).

McLendon, L. L., & Polis, K. (2007). The state adult edu-cation sta guide or going to scale: A guide or planning,implementing, and evaluating program improvement ini-tiatives. Washington, DC: The National Adult EducationPro essional Development Consortium. Online at: www.

naepdc.org/GTScorrected3.11.doc .Summerville, G., & Raley, B. (2009). Laying a solid oun-dation: Strategies or e ective program replication. Phila-delphia: Public/Private Ventures. Online at: www.ppv.org/ ppv/publications/assets/298_publication.pd .

Uvin, P., & Miller, D. (1996). Paths to scaling-up: Alter-native strategies or local nongovernmental organizations.Human Organization, 55 , 344354.

and to demonstrate success to existing stakeholders. Thus, thesepositive outcomes can be help ul in generating additional undingand other resources to scale the intervention even urther.

6. Share promising practices and lessons about scale with other nonpro ts.Scale-up is a relatively new concept in the nonpro t sector. Theo-ries and practices related to scale-up in the nonpro t world arestill emerging as those involved work to identi y best practices.Although some o the rameworks and lessons rom scale can beadapted rom the business world, the nonpro t world has its own

&> t h e o r y p r a c t i c eset o opportunities and challenges that a ect the scaling process,not the least o which is the act that the motivations or scalingare o ten less straight orward than in the business world, wherepro t is usually the primary motive. The more nonpro ts are will-ing to take the risk to go to scale, and work to overcome thechallenges in doing so, the more opportunities will be availableto identi y and adapt promising practices in the scaling processspeci c to the needs o nonpro ts.Erin HarrisProject Manager, Harvard Family Research Project Email: [email protected]

Key Readings on Scale
http://www.ssireview.org/articles/entry/going_to_scale/http://www.ssireview.org/articles/entry/going_to_scale/http://www.bridgespan.org/LearningCenter/ResourceDetail.aspx?id=398http://www.bridgespan.org/LearningCenter/ResourceDetail.aspx?id=398http://www.brfny.org/downloads/BlueRidge-TaprootFoundationCaseStudy.dochttp://www.brfny.org/downloads/BlueRidge-TaprootFoundationCaseStudy.dochttp://www.caseatduke.org/knowledge/scalingsocialimpact/bibliography.htmlhttp://www.caseatduke.org/knowledge/scalingsocialimpact/bibliography.htmlhttp://muve.gse.harvard.edu/rivercityproject/documents/final_scale_article_Clarke.pdfhttp://muve.gse.harvard.edu/rivercityproject/documents/final_scale_article_Clarke.pdfhttp://gse.berkeley.edu/faculty/CECoburn/coburnscale.pdfhttp://gse.berkeley.edu/faculty/CECoburn/coburnscale.pdfhttp://www.seedinit.org/download-document/13-seed-iisd-report-on-scale-up-and-replication-280708sml.htmlhttp://www.seedinit.org/download-document/13-seed-iisd-report-on-scale-up-and-replication-280708sml.htmlhttp://www.seedinit.org/download-document/13-seed-iisd-report-on-scale-up-and-replication-280708sml.htmlhttp://www.emcf.org/pdf/gcap_progressreportOct08.pdfhttp://www.emcf.org/pdf/gcap_progressreportOct08.pdfhttp://www.courtinnovation.org/_uploads/documents/Lessons.pdfhttp://www.courtinnovation.org/_uploads/documents/Lessons.pdfhttp://www.naepdc.org/GTScorrected3.11.dochttp://www.naepdc.org/GTScorrected3.11.dochttp://www.ppv.org/ppv/publications/assets/298_publication.pdfhttp://www.ppv.org/ppv/publications/assets/298_publication.pdfmailto:[email protected]:[email protected]://www.ppv.org/ppv/publications/assets/298_publication.pdfhttp://www.ppv.org/ppv/publications/assets/298_publication.pdfhttp://www.naepdc.org/GTScorrected3.11.dochttp://www.naepdc.org/GTScorrected3.11.dochttp://www.courtinnovation.org/_uploads/documents/Lessons.pdfhttp://www.courtinnovation.org/_uploads/documents/Lessons.pdfhttp://www.emcf.org/pdf/gcap_progressreportOct08.pdfhttp://www.emcf.org/pdf/gcap_progressreportOct08.pdfhttp://www.seedinit.org/download-document/13-seed-iisd-report-on-scale-up-and-replication-280708sml.htmlhttp://www.seedinit.org/download-document/13-seed-iisd-report-on-scale-up-and-replication-280708sml.htmlhttp://www.seedinit.org/download-document/13-seed-iisd-report-on-scale-up-and-replication-280708sml.htmlhttp://gse.berkeley.edu/faculty/CECoburn/coburnscale.pdfhttp://gse.berkeley.edu/faculty/CECoburn/coburnscale.pdfhttp://muve.gse.harvard.edu/rivercityproject/documents/final_scale_article_Clarke.pdfhttp://muve.gse.harvard.edu/rivercityproject/documents/final_scale_article_Clarke.pdfhttp://www.caseatduke.org/knowledge/scalingsocialimpact/bibliography.htmlhttp://www.caseatduke.org/knowledge/scalingsocialimpact/bibliography.htmlhttp://www.brfny.org/downloads/BlueRidge-TaprootFoundationCaseStudy.dochttp://www.brfny.org/downloads/BlueRidge-TaprootFoundationCaseStudy.dochttp://www.bridgespan.org/LearningCenter/ResourceDetail.aspx?id=398http://www.bridgespan.org/LearningCenter/ResourceDetail.aspx?id=398http://www.ssireview.org/articles/entry/going_to_scale/http://www.ssireview.org/articles/entry/going_to_scale/

8/9/2019 b09c074129f1943b4a172d23fb8542b5

7/24

8/9/2019 b09c074129f1943b4a172d23fb8542b5

8/24


Scaling Social Entrepreneurial Impact: The SCALERS Model

Paul N. Bloom and Aaron K. Chatterji o Duke University dis-cuss a model that they have developed to conceptualize scaling impact or social entrepreneurs.

Scaling social impact has become a major challenge or socialentrepreneursindividuals who start up and lead neworganizations or programs to address social problems usingchange strategies that di er rom those used in the past. Socialentrepreneurs who achieve initial success with their ideas o tenhave di culty replicating these ideas on a larger scale.

To determine what actors contribute to success when socialentrepreneurs scale up their e orts, we developed a model thatidenti es seven organizational capabilities, or drivers. We alsoindicate situational contingencies that might lead some drivers tobe more e ective in certain situations than in others. The driversare identi ed by the acronym SCALERS: sta ng, communicat-

ing, alliance building, lobbying, earnings generation, replicating,and stimulating market orces.Sta ng re ers to the organizations e ectiveness at lling its

labor needsincluding managers, sta , and volunteerswithpeople who have the right skills or their positions. Organiza-tions that prioritize this driver pay close attention to their person-nel and human resource unctions, so that recruiting, training,appraising, and compensating sta are done competently. Manyorganizations must pay equal or greater attention to recruiting,training, and managing unpaid volunteers, who are o ten the li e-blood o cash-starved social organizations. Boards o directorsneed to be adept at identi ying, recruiting, guiding, and retainingtop management to lead the organization.

Communicating re ers to the organizations ability to convincestakeholders that its strategy is worth adopting or supporting.Placing a priority on this driver means the organization is suc-cess ul at persuading potential bene ciaries to take advantageo its services or to change their behaviors in socially bene cialways, convincing volunteers and employees to work or the orga-nization, encouraging consumers to patronize the organizationsincome-generating activities, persuading donors to provide undsto the organization, or creating avorable public attitudes towardthe organizations programs.

Alliance building is the e ectiveness with which the organiza-tion has orged partnerships and other linkages to bring aboutdesired social changes. Recent research has identi ed alliancebuilding as an essential ingredient in success ul scaling. Organiza-tions that employ this capability e ectively do not try to do things

on their own, but instead orge uni ed e orts.Lobbying is the organizations ability to advocate or gov-

ernment actions that may work in its avor. 1 Organizations that

lobby skill ully succeed in getting courts, administrative agencies,legislators, and government leaders to help their cause.

Earnings generation re ers to the e ectiveness with whichthe organization generates a stream o revenue that exceeds itsexpenses. Organizations that are success ul at generating earningsdo not have trouble paying their bills or unding their activities.

Replicating re ers to an organizations e ectiveness in repro-ducing its programs and initiatives. An organization that is adeptat replication ensures that its services, programs, and other e ortscan be copied or extended without a decline in quality; training,

ranchising, contracting, and other tools are used to ensure qual-ity control.

Stimulating market orces covers an organizations ability to cre-

ate incentives that encourage people or institutions to pursue pri-vate interests while also serving the public good. An organizationwith this capability is success ul at creating markets or o erings(i.e., products and services) such as microloans or carbon credits.Stimulating market orces can lead to signi cant social change.

The SCALERS model proposes that the extent to which an indi-vidual driver infuences scaling success depends on various actorsin an organizations internal and external environment that canenhance or suppress a drivers infuence. For example, the degree towhich sta ng drives scaling success depends on the organizationslabor needs. When labor needs are high (e.g., the organization pro-vides highly skilled services to large numbers o clients), sta ng iscrucial or success ul scaling. However, when labor needs are low(e.g., the organizations strategy is not based on service delivery),other drivers are more critical to success. In some situations, e ec-tive deployment o all seven drivers is needed or success ul scaling.In others, success is dependent on a ew critical drivers.

Taken as a whole, the SCALERS model o ers a roadmap orsocial entrepreneurial organizations interested in scaling theirimpact. In addition, it can act as an evaluation ramework thathelps social entrepreneurs to track and assess scaling progress andto identi y ways to improve. For example, an organization mightuse the model to assess its per ormance and determine how pastactions have helped or hurt its ability to scale. The model canalso appraise the potential impact o uture plans. And, nally, anorganization might use the model to examine its ecosystem anddetermine how situational contingencies a ect the seven drivers,both negatively and positively.Paul N. BloomAdjunct Pro essor o Social Entrepreneurship and Marketing Email: [email protected]

Aaron K. ChatterjiAssistant Pro essorEmail: [email protected]

Fuqua School o BusinessDuke University

1. The term lobbying is used loosely here and does not re er only to registeredlobbyists, which could jeopardize an organizations tax-exempt status.

> b e y o n d b a s i c t r a i n i n g

Related Resources

Bloom, P. N., & Chatterji, A. K. (2009). Scaling social entre-preneurial impact. Cali ornia Management Review, 51 (3),114133.

Bloom, P. N., & Chatterji, A. (2009). Scaling social entrepre-neurial impact . Boston: Harvard Business Publishing.
mailto:[email protected]:[email protected]:[email protected]:[email protected]

8/9/2019 b09c074129f1943b4a172d23fb8542b5

9/24


Save the Childrens Literacy Programs in Rural America:Evaluation That In orms Scale-Up

Elizabeth Reisner o Policy Studies Associates discusses how thelearning gains o a childrens literacy program relate to the pro-

grams scaling process.

Amajor goal o Save the Children (STC) in the United Statesis to improve the literacy skills o children who live in thenations poorest rural communities, including locations inAppalachia, the Southeast, the Mississippi River Delta, the Gul Coast, the Southwest, and Cali ornias Central Valley.

The organizations literacy program began in 2003 with 1,800children in 15 pilot sites. These sites implemented the programsa ter school literacy model, which at that time emphasized guidedindependent reading practice and employed leveled books (i.e.,books organized by their degree o di culty),computer-assisted quizzes (i.e., quizzes admin-istered on a computer), and periodic assess-ment using a standardized, norm-re erencedreading test.

Also in 2003, STC hired Policy Studies Associ-ates, a Washington, DC research rm, to launcha program evaluation. STC asked us to measureprogram participants change in literacy skillsand to analyze these results in light o partici-pants di erences in program attendance, base-line literacy skills, and grade in school.

Because we initiated the evaluation just asSTC began program implementation, we hada chance to recommend program eatures that

would acilitate evaluation. In particular, weencouraged STC to design its attendance database to record thenumber o books that each child read as part o the literacy pro-gram and each childs scores on quizzes taken a ter reading eachbook. We also helped design the attendance database so that itcould be merged with records o each childs scores on the pro-grams reading test.

Using these data, we have produced a series o six annualevaluation reports that describe the baseline educational charac-teristics o participating children, their program attendance, theirlevel o engagement in the program (measured by books read,quizzes taken, and quiz scores), and their change in reading pro -ciency (measured in comparison with the reading test publishers

national norming group). We have also produced tailored annualreports or each program site and or clusters o sites with sharedcharacteristics.

By the end o the 200809 program year, the initiative hadgrown to 146 sites serving almost 15,000 children. In addi-tion to guided independent reading practice, the program nowincludes skill-building tutorials as well as activities to improvefuency and vocabulary, any o which can be o ered in school ora ter school.

Evaluation ndings show that 60% o 200809 participantsincreased their literacy per ormance by at least 2 NCEs (normal

curve equivalents) over and above the learning gain that wouldtypically result rom another year in school. Participants averagliteracy gain was 5.8 NCEs beyond the typical expected annualgain. Among children who participated in the program 55 days ormore in 200809, the proportion improving by at least 2 NCEswas 63%. Among children who attended 55 days or more andwho scored below grade level at the all 2008 baseline, 66%gained 2 or more NCEs. Higher numbers o books read and quiz-zes passed were statistically associated with higher NCE gains.

Although these ndings are subject to selection bias becausechildren attended the a ter school sessions voluntarily, they clearlysuggest that the literacy program was making a di erence or paticipating children at the same time that it was scaling up. These

observations prompt a question: Did evidence oparticipant learning infuence the growth and urther development o the program?

For the Evaluation Exchange, I asked Sprogram leaders to describe how program growthand documented learning gain intersected, i all. Mark Shriver, STCs vice president or U.programs, said that the relationship was positive and power ul. When he meets with educators and policy leaders to promote the program,he said, evaluation ndings are what he talkabout a ter he says hello. In the current tougeconomic climate, according to Shriver, state andlocal leaders will not consider any new education initiative unless it provides explicit evidenc

o e ectiveness. Andrew Hysell, STCs assocvice president or policy and advocacy, said that educators andpolicy leaders who make program decisions want to see evalu-ation evidence that is both rigorous and ocused on state educa-tional priorities such as literacy.

Ann Mintz, STC education advisor or U.S. programs, haanother perspective on the relationship o evaluation and scale-upFor her, evaluation is an essential tool in tracking program e ectiveness, determining what works best in the program, and pro-viding in ormed eedback to improve the program. For examplebased on evaluation ndings about which children bene t most,STC urges program leaders to recruit and retain children who tthe literacy programs individual success pro le (e.g., struggling

readers who are capable o regular attendance and active engage-ment in the programs literacy activities). Because regular pro-gram attendance appears to be so important in producing gains,STC encourages programs to educate parents about the value oregular attendance and to create authentic ways o encouragingattendance and personal engagement in program opportunities(no pizza parties or this health-conscious organization). According to Mintz, rewards or high attendance and engagement inSTC programs include gi ts o new books, the chance to read tkindergarten students, and the opportunity to eat lunch with theschool principal.

> e v a l u a t i o n s t o w a t c h

In the current tough

economic climate, state

and local leaders will

not consider any new

education initiative

unless it provides

explicit evidence o

e ectiveness.

continued on page 2

8/9/2019 b09c074129f1943b4a172d23fb8542b5

10/24


Lessons rom Evaluators Experiences with Scale

Heidi Rosenberg o Harvard Family Research Project and HelenWestmoreland o the Flamboyan Foundation spoke with several evaluators and program administrators about their experience

evaluating the process o going to scale.

H arvard Family Research Project spoke with three evalu-ators (see box below) to discover how evaluation canin orm and assess scaling e orts. They shared lessonsrom their experiences in evaluating programs as they went to

scale. Themes that emerged rom these conversations ollow.

The ways in which programs think about scale are driven inpart by their particular characteristics and activities.Scaling takes di erent orms, rom replication o an establishedprogram model, to a deepening o program e orts to achievemore measurable impact, to scaling components that have dem-

onstrated success. The approach taken depends in large part onthe needs and goals o the program being scaled.Replication. The replication model is traditionally used in the

business sector and has been adapted or the nonpro t sector. Inthe case o Citizen Schools, the organization wished to scale itsa ter school program across the country with the goal o aith ullyreplicating the original program model. Based on PSAs evaluationo the original program, which demonstrated its positive impacton student achievement, leadership at Citizen Schools elt the pro-gram as a whole was ready to expand beyond the home base.

Deepening o program e orts. Another way o raming scalinge orts relates to the depth o a programs impact and reach. TheIowa PIRC team took this approach: Scaling involved re raming

the programs e orts rom simply engaging in a wide variety o parent involvement activities to ocusing on the degree to whichthe PIRC showed evidence o impact along a set o meaning uloutcomes. In this case, scaling related to the depth o programimpact. The PIRC team reported that its early e orts sought toraise parent awareness, but that program administrators becamedissatis ed with the lack o measurable impact. As Mirr com-mented, Our scaling went rom a shallow understanding o parent involvement activities to committing to change in depthover time.

Identi cation o core components. Not all elements o a pro-gram can, or should, be scaled, since program components areo ten context-speci c and do not translate well to di erent loca-

tions. In these instances, the most promising practices are identi-ed and selectively scaled. SRIs evaluation o the GLOBE Programidenti ed success ul pro essional development practices amongwidespread, highly variable program sites; these practices werethen scaled across the program, thus increasing the number o sites that adopted these components. Penuel noted, The GLOBEProgram had wide variability in the success o its local partners.As part o our evaluation, we went in to help them understand,through comparative case study analyses o success ul partners,the actors o pro essional development that were associated withhigher levels o implementation.

Di erent phases o the scaling process require di erent evaluation approaches.To evaluate scale e ectively, researchers need to choose methodsand approaches based on the stage o the scaling process anddetermine how the data will be most use ul to the program ateach stage. During preparation, evaluation can help identi y whatprogram elements need to be in place in order to scale. Evalua-tion e orts may shi t depending on the scaling approach selected.Once programs have gone to scale, researchers can use evaluationdata to assess how well the program has scaled.

Using evaluation data to decide what elements to scale. Base-level evaluation data can help researchers identi y the programelements that must be in place or success ul program expansion.Reisner used the initial Boston-area impact study to identi y thekey components o Citizen Schools that a ected student success.Program sta analyzed the contexts and resources necessary tocreate those core components and built that understanding intoa consideration o where to implement program services. SRIsPenuel reported that evaluating widely scaled, eld-building pro-gram e orts requires identi ying the common elements underlyingprogram success: It is not particularly in ormative to say sim-ply that the context matters and that the context is messy. I weare thinking about eld building in this area, we have to assumethat there are some regularities across contexts that matter. Forexample, it matters that teachers perceive that the pro essional

> a s k t h e e x p e r t

Researchers Interviewed

Elizabeth Reisner, Policy Studies Associates (PSA)Evaluator o Citizen Schools, which sponsors a ter

school education programs or disadvantaged middleschool youth. PSA conducted an impact study evaluationo the original program in Boston and an implementationevaluation o the national program.

William Penuel, Center or Technology, SRI InternationalEvaluator o the GLOBE Program, an international

program in earth science and earth science education.SRIs evaluation ocused on the e ects o state policiesand pro essional development on the implementation o the science curriculum.

Robin Galloway, Research Institute or Studiesin Education, Iowa State University

Evaluator o the Iowa Parent In ormation and ResourceCenter (PIRC), a ederally unded program that providesregional and statewide services to promote amily engage-ment in education. Ed Redalen , Iowa PIRC programdirector, and Ron Mirr , a consultant to the Iowa PIRC,also contributed.

8/9/2019 b09c074129f1943b4a172d23fb8542b5

11/24


development and the program that you are trying to implement iscoherent with the school and district and state goals or learning.Once researchers have identi ed those regularities, or commonelements, Penuel pointed out, the next step is to ask, How doI measure that? And how do I t a model that goes across manyschools, in order to try to test my conjecture about the importanceo that actor?

Tailoring evaluation approaches to the scaling strategy. InIowa, the PIRC evaluation started by measuring program inputs(i.e., the services and resources provided). A ter deciding to shi ttheir emphasis toward deepening their services, PIRC administra-tors and the evaluator examined their qualitative data to identi ychampion teams within their Sustaining Parent InvolvementNetwork (SPIN) and targeted those sites or the initial scaling.This shi t rom counting program inputs to assessing depth o impact required program administrators to work closely withtheir evaluator; Galloway held requent meet-ings with the PIRC team to help its membersplan how their proposed scaling e orts andpoints o impact could be operationalized andevaluated to show e ect.

Using evaluation data to assess the scaled program. Once a program has gone to scale,the evaluation activities shi t to examining howthe scaling process is working. The implemen-tation study o the expanded national CitizenSchools program allowed the organization toassess implementation consistency across sites and to develop amore cohesive sense o program identity. The positive ndings

rom both the impact study o the initial site and the ollow-upstudy o the scaled program encouraged Citizen Schools to con-sider launching a rigorous experimental design impact study. As

Reisner noted, Our evaluation describes the similarity and diver-sity across sites, and the news is pretty good in terms o the con-sistency o implementation nationally. I the program just variedall over the map, then it would be premature to conduct a veryrigorous impact evaluation o the program.

E ective data collection systems are crucial to success ul scaling and evaluation.Reisner and the Iowa PIRC team discussed the importance o strong data collection e ortssites must be willing to provide thenecessary data, and evaluators need an appropriate in rastructureto gather and disseminate data. As Reisner observed, Getting thecooperation o the school system or data collection and researchis a nontrivial matter. It is much easier to do it on the ront endthan to go back and have to ll in a ter the act.

Structures and norms or collecting data are important or anyevaluation, but are especially critical or scaling e orts. Properdata collection and documentation help lay the oundation toassess a programs progress and reveal which elements o programsuccess should be scaled.

Programs need to ensure that their sites are collecting the samecore set o data in a coordinated manner. The Iowa PIRC teamsscaling e orts, or example, involved a shi t rom disparate datacollection methods among SPIN sites to procedures that everyone

in the network ollowed. Mirr portrayed it as a shi t rom essentially independent or no data to common data that everybodycollects the same way.

Scaling requires signi cant time and resources that may entail compromises or even sacri ces in other areas.SRIs Penuel noted a tendency to view educational and social ser-vice scaling as a process akin to corporate so tware start-upscaling models, where costs are concentrated on development,and dissemination or replication becomes much less costly andresource-intensive over time. In my view, educational scaling isnot at all like so tware; it is like services. I you really want somthing to scale, there is no point at which it becomes a whole lotless resource-intensive.

In Penuels view, this resource-intensiveness creates a tension between the demands o policy makerswhose support c

impact programs unding and scaling oppotunitiesand the realities o care ul, authenscaling o educational programs: Policymake

are always looking or ways to scale at low coand the point at which they are going to be ableto completely trans er ownership o the innovtion to the eld o practice; in only rare casedoes that ever happen. And usually it happenin cases where there is an easy t to the contextAnd i we are trying to make big changes,easy t to the context is not going to have

dramatic e ect on practice. The things that have the potential tochange are game-changersthey push against the main culture oeducation. They do things really di erently and there ore requiongoing e orts to organize.

Noting that scaling requires signi cant resources, Reisne

pointed out that success ul evaluations and program scalinge orts can lead to di cult choices. Citizen Schools is in the prcess o determining how they are going to move orward in termo the next phase o impact evaluation. The sequencing o thprograms evaluations thus ar has, as Reisner put it, positionedCitizen Schools to be able to make some very di cult choices itheir investments. On the one hand, they would probably like to

continued on page 2

> a s k t h e e x p e r

Related Resources

Penuel, W. R., Fishman, B. J., Yamaguchi, R., & Gallagher,L. P. (2007). What makes pro essional development e ective?Strategies that oster curriculum implementation. AmericanEducational Research Journal, 44 (4), 921958.

Vile, J. D., Arcaira, E., & Reisner, E. R. (2009). Progresstoward high school graduation: Citizen Schools youth out-comes in Boston. Washington, DC: Policy Studies Associ-ates. Available at: www.policystudies.com/studies/youth/ Citizen%20Schools%20Phase%20VI%20Report.pd .

I you really want an

education program to

scale, there is no point

at which it becomes less

resource-intensive.
http://www.policystudies.com/studies/youth/Citizen%20Schools%20Phase%20VI%20Report.pdfhttp://www.policystudies.com/studies/youth/Citizen%20Schools%20Phase%20VI%20Report.pdfhttp://www.policystudies.com/studies/youth/Citizen%20Schools%20Phase%20VI%20Report.pdfhttp://www.policystudies.com/studies/youth/Citizen%20Schools%20Phase%20VI%20Report.pdf

8/9/2019 b09c074129f1943b4a172d23fb8542b5

12/24


&> q u e s t i o n s a n s w e r sA conversation with

Mike Smith

Marshall Mike Smith is senior counselor to the secretary as well as director o

international a airs at the U.S. Department o Education. His previous positions in

government have included chie o sta to the secretary or education and assistant

commissioner or policy studies in the O fce o Education under the Carter admin-

istration and undersecretary and acting deputy secretary or education in the Clinton

administration. Dr. Smith has been a pro essor at Harvard University, the University

o WisconsinMadison, and Stan ord University, where he was also dean o the School

o Education. Most recently, he was program director or education at the William and

Flora Hewlett Foundation. His distinguished career includes positions on many national

commissions and panels, and he has authored numerous publications. Dr. Smith earned

his masters and doctoral degrees rom the Harvard Graduate School o Education.

Q What does the phrase going to scale mean?

AIn the current education environment, going to scalemeans taking a promising innovation and replicating it ina large number o places. Going to scale at a signi cant

level means spreading an innovation throughout an entire geo-graphic region. In the policy environment, going to scale meanstaking an idea that seems to work in a particular setting or inmultiple settings, codi ying it, and then en orcing it through stateor ederal legislation.

Q Why did the idea o scale come into the policy conversa-tion in education?A

Policymakers were rustrated that promising innovationsand good ideas were not spreading. I remember Presi-dent Clinton asking, Why dont ideas

travel? What he meant was, Why arentgood ideas replicated in other settings? Edu-cation re orms in this country are like refiesin a eldthe refies blink on and o , butthey are isolated and uncoordinated, so theydo not give o a concentrated or meaning ul

glow. Also, when things are tried and they ail,we have a tendency to try the same thing overagain later. Or we try re orms that work or awhile, but we do not support them long termand they go away.

In response to this rustration, there hasbeen a growing sense that we should stop start-ing rom scratch in education re orm and build more on what hasworked. This is where the idea o scale comes in. I we can learnhow to bring good ideas to scale, then we can start to make moreprogress in re orming education.

Q How do you know when an intervention is ready to goto scale?

AFor an intervention to go to scale, it must have exter-nal validity it must have similar e ects in a variety o contexts. There are several considerations in determining

external validity.First, the intervention needs a strong theory o change or

logic model that identi es the causal drivers necessary to producethe interventions outcomes. As part o this model, it is important

to identi y the contextual variables that can impact, positivelyor negatively, the success o those causal drivers. For example,an intervention that involves a reduction in school class sizewill probably not improve student achievement without addi-tional high-quality teachers to support that reduction. A goodunderstanding o the interventions causal drivers and the envi-

ronmental conditions required or them to besuccess ul leads to a good understanding o thecontexts in which that intervention can andshould be applied.

Second, the intervention needs to demon-strate large e ects. I the intervention takesplace in an ideal context, and the e ects

are relatively small, when the intervention ismoved to another location where conditionsare not optimal, the e ects will be overcome bycontext. The e ects need to be large enough toprevail in spite o relatively minor contextualconditions. To show large e ects, interventionstypically must introduce substantial changes in

the status quo. For example, studies o school-choice interven-tions in which students have options to attend di erent schools(e.g., through vouchers, charters, or magnet schools) o ten showlittle improvement in student achievement. This small impact is

Education re orms in thiscountry are like refies ina eldthe refies blinkon and o , but they are

isolated and uncoordinated,so they do not give o

a concentrated or meaning ul glow.

8/9/2019 b09c074129f1943b4a172d23fb8542b5

13/24


&> q u e s t i o n s a n s w e r slikely due, at least in part, to the act that students new schoolsare not di erent enough rom their old schools on importanteducational dimensions like curricula, teacher quali cations andbackgrounds, and school hours.

Third, fexibility must be built into the intervention. Inter-ventions do not operate uni ormly across di erent sites. To e ec-tively go to scale, the intervention must allow or some fexibilityand adaptation. The challenge is determining how much adapta-tion to allow be ore the intervention is too di erent rom the orig-inal model to be con dent that it will still show e ects. E ectiveinterventions typically balance structureand fexibility. For example, instruc-tional re orms in the Long Beach schooldistrict in Cali ornia are supported bycontinuous improvement eedback loopsthat provide in ormation about how andwhen to adapt the re orms to meet localneeds. Similarly, the Knowledge Is PowerProgram (KIPP) Academies, which pro-vide charter middle schools, use a set o standard operating principles known asthe Five Pillars, but let individual prin-cipals decide how to implement them intheir schools.

QOnce you have determinedan intervention is ready to goto scale, what challenges are

involved in bringing it to new settings?

AInterventions encounter several challenges when they goto scale. For example, an intervention will ace greaterresistance in settings where it is seen as disruptive to

the existing system or when it is replacing an existing interven-tion. In such cases, more e ort may be needed to convince thoseinvolved that the new intervention is worth adopting. When donewell, these disruptive interventions can cause power ul changesthat lead to sizable improvements.

Related to this, the interventions salience, duration, and inten-sity are especially important when replacing an existing interven-tion. To make the case or the new intervention, it needs to beseen as a signi cant improvement over the old approach. There-

ore, the new intervention needs to show large and stable e ects,which generally requires that it be conducted over a long periodo time.

Also, bringing in a new intervention o ten requires relearn-ing e ective practices developed under the old intervention. Forexample, teachers become more com ortable with and better atteaching a curriculum as time goes on. Without even thinkingabout it, they do continuous improvementthey build on whatthey did and learned the previous year. With a new curriculum,they have to relearn the whole system and develop a new seto habits.

Finally, as I said earlier, implementation may look and behavequite di erently rom one place to another. I the model is su -

ciently fexible, this variation should not be problematic. How-

ever, these contextual di erences may mean that it takes mortime and e ort to see expected results.

Q What evaluation strategies should accompany the processo going to scale?A

First, evaluations o the original intervention are needeto ensure that it has internal validitythat the interven-tions causal drivers are working as intended. This process

involves collecting implementation data to determine that the pro-gram is implemented as planned, as weas outcome data to ensure that the program achieves what it set out to achievI you have that, then you know i tintervention does not work in a new setting, it is likely due to contextual diences that a ect implementation raththan a faw in the model itsel .

Second, once the original site hbeen evaluated or internal validity anit goes to scale in new locations, evaluations are needed in the new sites to heldetermine what tweaks are necessary taccount or di erences in context.I mentioned earlier, evaluations at thistage should emphasize ongoing learing and continuous improvement tensure the scaling process is as succes

ul as it can be.

Heather B. Weiss, Ed.D.Director, Harvard Family Research Project Email: [email protected]

Helen Janc MaloneGraduate Research Assistant, Harvard Family Research Project Email: [email protected]

Mike Smith

Related Resource

Smith, M. S., & Smith, M. L. (2009). Research in the policyprocess. In G. Sykes, B. Schneider, & D. N. Plank (Eds.),Handbook o education policy research (pp. 372397). NewYork: Routledge Publishers ( or the American Educational

Research Association).This article explores the relationship between educationresearch and policy. In response to the impression that manyeducation policies and interventions have little impact oneducation outcomes, the authors examine ways to improvethe quality and use ulness o education research. The arti-cle concludes with recommendations or policymakers andpolicy researchers, as well as suggestions or innovations tohelp produce major improvements in student achievementoutcomes.

8/9/2019 b09c074129f1943b4a172d23fb8542b5

14/24


Developmental Stages or Evaluating Scale

Sarah-Kathryn McDonald o the University o Chicago describesa conceptual model designed to demonstrate the role o evalua-tion in the scale-up process.

Promising education interventions typically go through adevelopmental process that starts with implementation ata single site, and then works incrementally toward the goalo scaling up their success ul practices to additional settings toimpact larger, more diverse populations. Evaluation can help anintervention proceed rom one stage to the next, with di erentevaluation strategies accompanying each step o the interven-tions development. The Data Research and Development Center,a research and technical center unded by the National ScienceFoundation as part o the U.S. Interagency Education ResearchInitiative, has developed a conceptual model that speci es theevaluation methods appropriate at di erent stages in the inter-

ventions development. The model describes ve evaluation stagesthat culminate in an interventions widespread dissemination andadoption. Ideally, an intervention should proceed through eachstage in order.

Stage 1: Proo o Concept . Stage 1 involves determining wheth-er an intervention is su ciently promising to develop and scale(including making any necessary improvements to the interven-tion). The goal is to produce evaluation data that can demonstratewhich parts o the intervention can accommodate fexibility andwhich parts are not negotiable.

Stage 2: Establish E cacy . This stage determines whether theintervention can achieve its intended results under ideal circum-stances. At this stage, it is crucial that the intervention be imple-mented (and evaluated) with the eatures and in the context thatare seen as optimal or success.

Stage 3: Demonstrate E ectiveness . Stage 3 aims to assesswhether an intervention will achieve its objectives outside theideal context measured in Stage 2. The objective is to establishwhether an intervention works in a real-world setting, with allo its complications.

Stage 4: Scale-Up and Test Again . Stage 4 aims to demon-strate the interventions impact when it is implemented amonglarger numbers o individuals across many contexts. This stagealso examines the contextual actors that may infuence the inter-ventions impact in di erent settings. In addition to identi yingreasons or any observed discrepancies, these data can provide

eedback to help re ne the intervention or develop guidelines toensure it operates as intended in particular contexts.

Stage 5: Postmarketing Research . The nal stage explores theollowing questions: (a) I an intervention demonstrated to work

at scale is then more widely adopted, what else do we need tolearn about its e ectiveness in additional contexts at larger scales?and (b) What can and should we seek to learn about the sustain-ability o its impact and relevance once a market has been satu-rated and the intervention becomes the status quo? Postmarketingresearch can enrich our understanding o how the impact variesamong slightly di erent populations. It can also help identi y thecircumstances under which it might make more sense to adapt the

intervention to the local context rather than adopt the originalelements o the intervention.

Throughout the scale-up process, the context o the interven-

tion, and both the original intervention and the new sites intowhich it is scaled, must be kept ront and center. Because everysetting has its own needs and circumstances, scale-up should notuncritically replicate the intervention. Rather, scale-up should bea context-based approach that is supported by evaluation at eachstage along the way.

Sarah-Kathryn McDonald, Ph.D., MBAExecutive DirectorCenter or Advancing Research and Communication in Science,Technology, Engineering, and MathematicsNational Opinion Research Center at the University o ChicagoEmail: [email protected]

The author would like to thank Barbara Schneider (principal investi-

gator o the Data Research and Development Center and the Centeror Advancing Research and Communication in Science, Technol-

ogy, Engineering, and Mathematics) or her many contributions to,and continued support or, this work. This material is based uponwork supported by the National Science Foundation under GrantsNo. 0129365 and 0815295. Any opinions, ndings, conclusions, orrecommendations expressed are those o the author and do not neces-sarily refect the views o the National Science Foundation.

> p r o m i s i n g p r a c t i c e s

Related Resources

McDonald, S . (2009) . Scale-up as a ramework or interven-tion, program, and policy evaluation research. In G. Sykes, B.

Schneider, & D. N. Plank (Eds.), Handbook o education pol-icy research (pp. 191208). New York: Routledge Publishers( or the American Educational Research Association).

This article discusses the ve stages o evaluation that edu-cational researchers commonly use as they design, conduct,and implement evaluation studies: proo o concept, establishe cacy, demonstrate e ectiveness, scale-up and test again,and postmarketing research. The article emphasizes the valueo considering the primary audiences or the evaluation (i.e.,who is interested in the results and who can bene t romthem) as well as what these audiences plan to do with theresults and in what time rame. The article also suggests howevaluators can disseminate and communicate their ndings inways that are likely to increase the chances that the data they

collect will enhance policy and practice.McDonald, S. K., Keesler, V. A., Kau man, N. J., & Schneider,B. (2006). Scaling-up exemplary interventions. Educational Researcher, 35 (3), 1524.

This article has three objectives. First, it articulates thegoal o scale-up research in the eld o education. Second, itdiscusses the importance o context in conducting scale-upresearch. Third, it o ers practical guidelines or study designsthat can help education researchers provide evidence that theimpacts measured can be applied to additional settings.
mailto:[email protected]:[email protected]

8/9/2019 b09c074129f1943b4a172d23fb8542b5

15/24


continued on page 2

> p r o m i s i n g p r a c t i c e

Roblyn Anderson Brigham and Jenni er Nahas o Brigham Nahas Research Associates discuss the implications o their evaluationo the Childrens Aid Society/Carrera Integrated School Model

or expansion o the model to new school settings.

For the past 25 years, Dr. Michael Carrera and his team romthe Childrens Aid Societys (CAS) Adolescent Sexuality andPregnancy Prevention Program have established a model ora ter school programs working with youth in middle and highschool. The model relies on a holistic, long-term approach topregnancy prevention and sexuality education that helps youthdevelop personal goals and the desire or a productive uture.With a designation by the Coalition or Evidence-Based Policyas a Top Tier program based on a 3-year random assignmentevaluation, CAS/Carrera has a demonstrated track record o suc-cess.1 CAS/Carrera is now engaged in an e ort to incorporatethe model into the school-day curriculum at public and charterschools in targeted locations nationwide. This approach, calledthe Integrated School Model, began in 2006 with one school andis now operating in ten schools.

The Integrated School Model provides the seven oundationalcomponents o the CAS/Carrera a ter school model during theschool day: daily education; weekly employment; weekly mentalhealth classes and daily services; weekly amily li e and sexual-ity education; sel -expression; li etime individualized sports; andno-cost, comprehensive medical and dental services. Unlike otherin-school programs having similar goals, the model does not sim-ply provide additional social work sta or serve only as a health

center or students. Rather, CAS/Carrera brings sta , resources,training, curricula, and expertise into partner schools and workswith teachers to provide the support students in low-incomeneighborhoods need to succeed in school and li e.

Critical Issues or Expanding the ModelTo understand early implementation o the model, CAS/Carreracommissioned Brigham Nahas Research Associates to conduct astudy to: (a) document how the model is implemented and integrat-ed into the school setting during the early phase; (b) understandimplementation strengths, challenges, and opportunities; and (c)document perceptions o early outcomes. The study involved sitevisits and in-depth interviews with the principals at each o our

schools, sta o the Regional Implementation Centers, and CAS/ Carrera sta working in each o the schools.Based on this evaluation, there is reason to be excited about

the possibility or expanding the model to other schools and com-munities. Findings rom the evaluation suggest several criticalissuesstrengths, challenges, and lessons learned rom the Inte-grated School Model pioneersthat can in orm CAS/Carrerasthinking as the organization moves orward:

Matching philosophies. Philosophical underpinning o themodel is clearly articulated, consistent, and non-negotiable. The

1. Coalition or Evidence-Based Policy. (2009). Two new social program modelsidenti ed as meeting top tier evidence o e ectiveness. Washington, DC: Author.

better the match between the CAS/Carrera philosophy and thato the school, the quicker and easier it is to integrate the modelinto the school.

Supportive and fexible school leaders. Ease o implementatdepends on the unwavering support o the school leader, a fexibleschool management structure, and the ability to extend schoolhours. It is important to look or these eatures in the schools thatadopt the model.

Needs o students. The model lls signi cant and widely idented service gaps in each o the schools. Principals see CAS/Carre

as a key to addressing challenges or which they have no othesolution and no other resources available. None o the schoolshad comprehensive mental health supports or students or a sexeducation curriculum in place be ore the arrival o CAS/Carrera

Flexible implementation. The content o the Integrated SchoModel is non-negotiable, but fexible implementation o the sevencore components shows that the model can be integrated intomany di erent kinds o settings (public schools, charter schooetc.), as long as leadership is willing to work with CAS/Carrera onoperational issues such as scheduling and hiring.

Early Evaluation to In orm Expansion o aTeen Pregnancy Prevention Program

Related Resource

CAS/Carrera Integrated School Model Evaluation Report

Brigham, R. A., & Nahas, J. (2008). Childrens Aid Society/ Carrera Integrated School Model: Documentation o earlyimplementation in our schools . Cambridge, MA: BrighamNahas Research Associates.

This report discusses the methods, ndings, and implica-tions o Brigham Nahas Research Associates evaluation o theCAS/Carrera Integrated School Model. The ndings includedata about implementation, which in ormed the lessons orscale discussed in this article, as well as perceived outcomes atthe student, classroom/teacher, and school levels.

At the student level, evaluators saw evidence o improvedconnections with adults, management o emotions, and sel -expression/communication. At the classroom/teacher level,data suggest that the program is contributing to calmer class-rooms and better management o student behavior. At theschool level, evaluators saw evidence o the ollowing: (a) ashi t in culture toward in using youth development into edu-cation; (b) aster interventions in crisis situations; (c) earlieridenti cation o students needs; (d) better parent communi-cation and connection; and (e) a stronger sense o belonging,cohesion, and school spirit in the cohorts served by the Inte-grated School Model.

For more in ormation on the CAS Adolescent Sexual-ity and Pregnancy Prevention Program, visit: www.stopteenpregnancy.com .
http://www.stopteenpregnancy.com/http://www.stopteenpregnancy.com/http://www.stopteenpregnancy.com/http://www.stopteenpregnancy.com/

8/9/2019 b09c074129f1943b4a172d23fb8542b5

16/24

8/9/2019 b09c074129f1943b4a172d23fb8542b5

17/24


Applying a Broader Concept o Scaleto Evaluate a Funding Strategy

Erin Harris and Priscilla Little o Harvard Family Research Proj-ect (HFRP) discuss how HFRP used a multidimensional concept o scale to evaluate The Atlantic Philanthropies Integrated Learn-ing Cluster strategy.

T he Atlantic Philanthropies developed its Integrated Learn-ing Cluster (ILC) strategy as a multiyear national e ort tokeep disadvantaged youth ages 8 to 16 engaged in learningduring out-o -school time (OST) hours. Recognizing that schoolsalone cannot meet the learning needs o our children, the strategywas based on the belie that extra support provided beyond theschool day helps to strengthen academic skills and increase emo-tional intelligence. To achieve these goals, Atlantic invested in arange o nonschool supportsgrants or direct service, in rastruc-ture, and advocacy e orts.

At the heart o the ILC strategy was the need to bring the OSTarena to scale to better serve disadvantaged children. But whatdoes scale really mean? Traditional de nitions ocus on expand-ing the number o people served through replication or adaptationo existing models and re orm e orts. The ILC strategy, how-ever, applied a notion o scale that moves beyond this de nitionto encompass a broader understanding, as de ned by CynthiaCoburn. 1 Her operationalization o scale has our components:spread, depth, sustainability, and shi t in re orm ownership.

As part o the evaluation o Atlantics ILC strategy, HarvardFamily Research Project used Coburns de nition to help iden-ti y strategic opportunities in Atlantics investments that couldenhance these our components o scale, thereby achieving the

goals o the ILC strategy. For example, by applying this scalingraming to the ILC grantees, we discovered that advocacy is a key

component in the scaling process and was used by most Atlanticgrantees whether or not they were unded to do so. As a result,HFRP recommended that Atlantic consider incorporating advo-cacy investments across all o their unding strands (rather thanonly a subset). We discuss the components and their applicationto the evaluation o the ILC strategy below.

Spread is the central notion o scale as traditionally de nedthe idea that an e ort will expand to increase the number o pro-grams and people served. In Coburns de nition, spread extendsbeyond merely upping the numbers to include the spread o ideas, belie s, values, and principles that support the e ort being

brought to scale. Atlantic invested not only in organizations thatcan expand their reach, but also in resources that help intro-duce evidence-based, best-practice in ormation into the nationaldialogue in order to build public and policy support or OSTprograms.

Depth re ers to the nature and quality o change. Accordingto Coburn, to be at scale, re orm e orts must e ect deep andconsequential change at the practice level. 2 Atlantics investmentstrategy devoted signi cant resources to OST program models

1. Coburn, C. (2003). Rethinking scale: Moving beyond numbers to deep and last-ing change. Educational Researcher, 32 (6), 312.2. Ibid., p. 4.

that had already achieved some early wins in their e orts toimpact practice. Further, scaling quality programs (i.e., those thhave undergone rigorous evaluation, have prevalent quality stan-dards, use strategic business plans to guide their growth, and arecommitted to reducing barriers to program participation) was acritical part o Atlantics overall strategy.

Sustainability re ers to the idea that scale has meaning ovetime. As Coburn suggests, the distribution and adoption oan innovation are only signi cant i its use can be sustained.Atlantics ILC investments, particularly in advocacy unding, werintended to promote and sustain high-quality OST experiences

or disadvantaged youth by building awareness o and supporor policies that increase ederal and state unding or e ec

OST programs. Atlantics early investments in organizationaldevelopment and capacity building increased the likelihood thatILC grantees would continue to be active participants in the OSTin rastructure long a ter Atlantics investments had ceased. Futher, most ILC grantees had multiple sources o private and cor-porate sponsorship, enabling them to leverage Atlantics resourcesto garner additional support.

Shi t in re orm ownership, as Coburn notes, re ers to shiing authority and knowledge rom external actors to those whoare either implementing re orm or building the capacity to doso. In the case o ILC investments, these external actors wernational-level OST advocacy organizations that pushed or policies to improve OST program access and quality. To ensure thesuccess o these policies, all OST stakeholders must be on boardwith and take ownership o these e orts at the ground level. A

noted above, many ILC grantees demonstrated this ownership byimplementing policy advocacy strategies and activities whetheor not they were unded to do so. This shi t came about as grantees realized that advocacy e orts were necessary to achieve thegoals and that they needed to become active advocates themselvesrather than rely solely on others to advocate or robust OST poli-cies. I this component o scale continues to progress among ILgrantees, then over time more grantees will internalize advocacy

unctions as part o doing business, and the national advocacyorganizations will build the capacity o other local-level OST programs to ollow suit.

Coburns broadened vision o scale allowed or a deeper, mornuanced understanding o Atlantics ILC strategy investments

Although the ramework was initially developed or educatiore orm, the concepts have applications across a range o eldand types o e orts.

Erin HarrisProject Manager, Harvard Family Research Project Email: [email protected]

Priscilla LittleAssociate Director, Harvard Family Research Project Email: [email protected]

3. Ibid., p. 6.

> p r o m i s i n g p r a c t i c e

8/9/2019 b09c074129f1943b4a172d23fb8542b5

18/24

8/9/2019 b09c074129f1943b4a172d23fb8542b5

19/24


Harvard Family Research Project describes the unctions o theWhite House O ce o Social Innovation and Civic Participationand its oversight o the ederal Social Innovation Fund.

In response to the perceived lack o innovation and use o rig-orous approaches to identi ying what works in the non-pro t sector, last year President Obama created the O ce o Social Innovation and Civic Participation. The o ce is a part o the Domestic Policy Council, the entity that coordinates domesticpolicy making in the White House.

In April 2009, Sonal Shah, who previously headed Googlesglobal development team, the search engines philanthropic arm,was appointed to lead the new o ce. Prior to Google, she hadworked or think tanks including the Center or Global Develop-ment and the Center or American Progress as well as or the U.S.government in the Department o Treasury.

The new o ce seeks to address the need to identi y and scaleup success ul nonpro t initiatives and has our primary goals.First, it aims to develop partnerships between the government andnonpro ts, businesses, and philanthropists. Second, it works tosupport the rigorous evaluation and scaling o innovative, prom-ising practices. Third, it supports the use o new media tools toencourage greater civic participation. Lastly, it promotes nationalservice.

The o ce emphasizes the value o government unding orbottom-up initiativesthat is, nonpro t initiatives that are drivenby those outside o government. Speaking at the 2009 Global Phi-lanthropy Forum Con erence, Shah told donors and grantmakers,Partnerships matter a lot to us. Its not just that the governmentdoes, and everybody else ollows. It really is about where can welearn and what can we do di erently. 1

The o ce welcomes ideas and innovations rom new andlong-standing organizations alike, as Melody Barnes, director o the Domestic Policy Council, stressed in an interview with theChronicle o Philanthropy: O ten when theres change, peoplebelieve we are leaving whats been working over time behind. Theidea is to add capacity and to bring new ideas and new peopleand new models to the ore and to the task o addressing these bigchallenges. This shouldnt be seen as the White House turning itsback on the more traditional allies and moving on to somethingnew. An organization thats been around or 80 years can havean innovative idea, an idea that will have signi cant impact andbuild capacity. 2

Social Innovation FundThe Social Innovation Fund is a priority or the new o ce. Theo ce is working in collaboration with the Corporation or

1. Wilhelm, I. (2009). White House Social Innovation o ce to have three goals.The Chronicle o Philanthropy. Retrieved January 14, 2010, rom: http://philanthropy.com/news/con erence/7957/white-house-social-innovation-o ce-to-have-three-goals .2. Perry, S. (2009). White House o cials discuss plans or Social-Innovation O ce. The Chronicle o Philanthropy. Retrieved January 14, 2010, rom: http://philanthropy.com/news/updates/8381/white-house-o cials-discuss-plans- or-social-innov ation-o ce.

White House O fce o Social Innovation and Civic Participation

National and Community Service, the ederal agency that runsnational service programs, to set up the Social Innovation Fundpilot program. This program was authorized in the Edward M.

Kennedy Serve America Act o 2009, which expanded the national service programs administered by the Corporation or Nationaland Community Service. According to Nicola Goren, acting CEOo the corporation, The goal is to build a pipeline o organizations and practices with strong evidence, and the capacity to growand increase the impact o their work. The Social Innovation Fundwill provide the support needed to help move organizations romthe promising stage to the stage where they have more concreteevidence that what they do works. 3

In the FY2010 budget, President Obama requested $50 mil-lion or the Social Innovation Fund program. The dra t Notice oFunding Availability or this und was made available to the public or review in December 2009. According to the press releas

or the Fund, every dollar in ederal unding must be matched bthree dollars in private unds. The plan is to provide 5-year grants,ranging rom $5 million to $10 million, to between ve and sevenintermediaries (i.e., grantmaking organizations) in FY2010. Theseintermediaries will then provide subgrants to community organi-zations to identi y and scale promising, results-oriented nonproprograms. In addition, the intermediaries that receive edera

unding are required to provide other supports (besides undingto the identi ed promising programs, including assistance withmanagement, undraising, and evaluation needs.

Harvard Family Research Project

3. Goren, N. (October 8, 2009). Replicating success. White House Blog . Retri January 14, 2010, rom: www.whitehouse.gov/blog/Replicating-Success.

> s p o t l i g h

Related Resources

The press release about the creation o the White HouseO fce o Social Innovation and Civic Participation can be

ound online at: www.whitehouse.gov/the_press_o ce/ President-Obama-to-Request-50-Million-to-Identi y-and-Ex pand-E ective-Innovative-Non-Pro ts.

In ormation on the Social Innovation Fund can be oundon the Corporation or National and Community Serviceswebsite at: www.nationalservice.gov/about/serveamerica/ innovation.asp . This website includes press releases, o cialstatements, and updates on the Fund, including in ormationabout the status o the Notice o Funding Availability.

Serve.gov is an online resource managed by the Corporationor National and Community Service to help citizens nd

volunteer opportunities and to acilitate the creation o newones. It is part o United We Serve, a national service ini-tiative. For more in ormation, visit: www.serve.gov. On thiswebsite, you can create your own project, recruit volunteers,

nd volunteer opportunities, and read and share stories o peoples experiences in volunteer service.
http://philanthropy.com/news/conference/7957/white-house-social-innovation-office-to-have-three-goalshttp://philanthropy.com/news/conference/7957/white-house-social-innovation-office-to-have-three-goalshttp://philanthropy.com/news/conference/7957/white-house-social-innovation-office-to-have-three-goalshttp://philanthropy.com/news/updates/8381/white-house-officials-discuss-plans-for-social-innovation-officehttp://philanthropy.com/news/updates/8381/white-house-officials-discuss-plans-for-social-innovation-officehttp://philanthropy.com/news/updates/8381/white-house-officials-discuss-plans-for-social-innovation-officehttp://www.whitehouse.gov/the_press_office/President-Obama-to-Request-50-Million-to-Identify-and-Expand-Effective-Innovative-Non-Profitshttp://www.whitehouse.gov/the_press_office/President-Obama-to-Request-50-Million-to-Identify-and-Expand-Effective-Innovative-Non-Profitshttp://www.whitehouse.gov/the_press_office/President-Obama-to-Request-50-Million-to-Identify-and-Expand-Effective-Innovative-Non-Profitshttp://www.nationalservice.gov/about/serveamerica/innovation.asphttp://www.nationalservice.gov/about/serveamerica/innovation.asphttp://www.nationalservice.gov/about/serveamerica/innovation.asphttp://www.nationalservice.gov/about/serveamerica/innovation.asphttp://www.whitehouse.gov/the_press_office/President-Obama-to-Request-50-Million-to-Identify-and-Expand-Effective-Innovative-Non-Profitshttp:

Documents

b09c074129f1943b4a172d23fb8542b5