Presentation at The Conference for Family Literacy

Louisville, KentuckyBy Apter & O’Connor Associates

April 2013

Evaluating Our Coalition: Are We Making an Impact?

Critical stakeholder mass

Greater efficienci

es

Broader goals

accomplished

Community benefits

Why a Coalition?

Evaluation can guide the change process

Evaluation can monitor the health of your coalition

Evaluation can “tell your story”

Evaluation can “dig deeper”

Evaluation can measure your success and answer the “so what” questions

Why Evaluate?

Effectiveness

Why Evaluate? Provide Accountability to

Community, Funders & Stakeholders

EffectivenessEffectiveness

Quality Efficiency

Effectiveness

Step 1: Defining YOUR Coalition

Cooperative Group

• Short term goals/tasks

• Linkages may be formalized but still advisory

Network• Dialogue• Informal

linkages• Information

exchange

Collaborative • Long-term

commitment• Formal agreements• Shared leadership

and decision making

• Shared community-wide indicators of success

Partnership• Share/merge

resources• Joint planning but

individualized authority

• Frequent communication

• Common issue

SO WHAT!! The Impact of Your Coalition

Step 2: Meeting Evaluation Challenges

1. Lack of clarity about the expected outcomes from the coalition: What to evaluate

2. Coalitions take time to build3. Coalitions are not static4. Measurement: Attribution vs.

Contribution (who gets “credit”)

Challenge #1: Lack of Clarity about Expected Outcomes

Some Solutions• Reflect back to your logic model:

reasonable, feasible, focused outcomes

• Clarity about what the coalition will bring to the community that individual programs cannot.

• The evaluation must match the level of anticipated outcome

Communities

Systems

Agencies/Organizations

Individuals/clients

Level of Impact

Individuals

Change in behaviors,

attitudes, skills

Agencies/Organizations

Change in behaviors, attitudes, knowledge and skills

Change number and type services provided

Change in policies, proceduresLeadership development

Systems

Change in behaviors, attitudes, knowledge and skills

Increase in coordination/collaboration/cohes

ion within or across systemsChange in service delivery

Joint decision making

Community

More resources/services for the community

Change in legislation, regulation, policy

Heightened community awareness/civic action

Demonstrated change in social, economic, environmental conditions

Challenge #2: Coalitions Take Time to Build

• Clarify the “development phase” of your coalition and the key tasks that need to be accomplished

• That will lead you to appropriate questions and important data to collect

Some Solutions

Evaluating the right thing at the right time!

• Planning and forming• Why do we need this

collaboration? • Who should be involved? • Do we have a common vision?

Formative Evaluation:--Readiness and capacity

“Developmental Evaluation”:Evaluating the Right Thing at the Right Time…

Developing • Are we on track? • Is our coalition meeting the needs of

members? • Are we achieving early outcomes?

Process Evaluation:-- Beginning to implement

Evaluating the Right Thing at the Right Time

Evaluating the right thing at the right time!

Maturing and Sustaining • What have we

accomplished?• What are the benefits for

our coalition and for whom?• What is the value of our

effort?Outcome/summative evaluation-- Accountability

Evaluating the Right Thing at the Right Time

Challenge #3: Service Systems and Communities Are Not Static

Solutions to enhance evaluation

• Revisit your logic model:– Are the new strategies are still

in line with your vision– Does the coalition membership

still include the “right” stakeholders

Some Solutions

Challenge #4: Attribution vs. Contribution

• How do we insure “self interest” of member agencies/organizations are met?

• How do we distinguish outcomes of the coalition vs. outcomes by programs funded by the coalition vs. other contextual factors?

Solutions to enhance evaluation

• Do a good job of collecting the right data– Reflects the coalition expected level of

impact– Reflects the coalition developmental

stage

• Critical importance of process to outcome: continue to measure perception of members

• Shared measurement systems

Some Solutions

Not everything that counts can be counted!

Phases of Shared Measurement System

STEP I Design

STEP 2Develop

STEP 3Deploy

•Agree on system and relation to logic model

•Review current state of knowledge/data

•Agree on governance structure

•Identify approach, metrics, confidentiality, etc.

•Develop platform and tools

• Test and refine platform/tools

•Staff for data management and synthesis

•Learning forums

•Ongoing support

•Review, refine, improve, ongoing evaluation of usability and impact

Source: FSG Analysis, www.fsg.org 2011

The COMET® System

• Clarity among organizations about WHY you are collecting this data and how it will be used

• Getting organizations to agree on a set of shared indicators that reflect their work

• Silo nature and reporting requirements of funders

• And………….

Cautions….

Outcome

Process

Formative

Measure infrastructure,

functioning and procedures of the coalition

Measure indicators of

extent of implementation

Measure change and realization of

vision

Evaluation Types are All Related….

SURVEYS: member benefits, perceptions of stakeholders, informants, community

INTERVIEWS/FOCUS GROUPS stakeholders, beneficiaries, community members

OBSERVATIONS: site visits, meetings

RECORDS, DOCUMENT REVIEW

CASE STUDIES: policy analysis, beneficiaries, build a story of the coalition’s work and impact

INDICATORS: statistical analysis for significance, trend data, change in targets, cost-benefits

EV

ALU

ATIO

N

Tools of the Trade

EV

ALU

ATIO

N

Collect information from SEVERAL SOURCES (community residents at large, targeted clientele, elected officials)

Collect information using SEVERAL METHODS (survey, focus group and site observations for perceptions)

Consider OUTSIDE EVALUATORS (to add credibility, expertise, objectivity)

Collect both QUANTITATIVE (the numbers) and QUALITATIVE(the story) data

USE your findings

Tools of the Trade

Pre-Conditions• Urgency for change

• Adequate financial resources

• Influential champions

Conditions for having impact• A common agenda (vision)• Continuous

communication• A shared measurement

system with a “short list” of indicators

• Mutually reinforcing activities (relating to vision

• A centralized backbone structure

Successful Coalitions

Kania and Kramer, Collective Impact, 2011 www.ssireview.org

 

1. Selection of a core set of community outcome indicators by the literacy coalition

2. Provide training and technical assistance in performance measurement

3. Assist community literacy programs to track and use their own outcomes

4. Provide support for community-wide outcome data collection

5. Provide support for analysis of the outcome information.

Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management for Community Literacy Coalitions, Washington, DC 20008

Contributions of a Literacy Coalition

 

6. Use the findings to help attract funding

7. Use the findings to identify and report community literacy program needs and literacy condition

8. Use the findings to help identify and disseminate successful (“best”) practices in your community and elsewhere

9. USE THE FINDINGS TO CELEBRATE GOOD LITERACY OUTCOMES

Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management for Community Literacy Coalitions, Washington, DC 20008

Contributions of Evaluation

For more information…

Literacy Powerline www.literacypowerline.com

Apter & O’Connor Associates

www.apteroconnor.comdianne@apteroconnor.com

cynthia@apteroconnor.com

For more information…The Literacy Powerline Site

www.literacypowerline.com or

The Apter & O’Connor Associates Site

www.apteroconnor.comor

Dianne@apteroconnor.comCynthia@apteroconnor.com