UNWOMEN/IPEN Transformative Mixed Methods Evaluation: Day 3 Mixed Methods

UNWOMEN/IPEN TRANSFORMATIVE MIXED METHODS EVALUATION:DAY 3 MIXED METHODS

Prof. Donna M. MertensGallaudet UniversityAlmaty, KazakhstanJuly 2011

IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

Three days togetherDay 1: What is the role of the evaluator?

Overview of evaluation (Terms of Reference)Transformative paradigmDiversity & social justice; privilegeCultural competence

Day 2: Quantitative & qualitative approaches

Day 3: Mixed methods approaches


Methods Options Quantitative approaches such as experimental,

quasi-experimental, causal-comparative, correlational, survey, and single-case designs

Qualitative approaches such as group processes (e.g., focus groups or some indigenous methods), case studies, ethnographic research, phenomenological research, and PAR

Gender analysis is a mixed-methods approach that provides a framework for transformative research and evaluation that has potential for transfer to other groups that experience discrimination.

Mixed methods are most likely to be the approach of choice because of the need to integrate community perspectives into the inquiry process, thus necessitating collection of qualitative data during the research or evaluation process. (Mertens, 2009, TRE, p. 165).


Mixed Methods Decisions Compatibility: Match the purpose, focus,

questions, and design. Timing: Determine the temporal

relationship between the quantitative and qualitative data collection.

Weight: Establish priority or emphasis of the qualitative and quantitative approaches in the study.

Mixing: Determine when quantitative and qualitative strategies/data will be mixed in the process of evaluation.


Concurrent DesignQuantitative and Qualitative occur more or less simultaneously

Qualitative Quantitative

Sequential Design: Quantitative Followed by Qualitative

OR

Sequential Design: Qualitative Followed by Quantitative

Quantitative

Qualitative

Mertens, 2009, TRE, p. 167


Community Participation


Example: Making VisibleBotswana youth:addressing

powerinequities in thefight against HIV/AIDS using atransformative

lens


Transformative Mixed Methods Design

Stage 4Concurre

nt

Stage 3Sequenti

al

Stage 2Concurre

nt

Stage 1Qual

Assemble team; read

documents; engage in dialogues

Preliminary

studies: youth, older

men

Process eval

Pilot intervention: Observations,

Interviews, Surveys

Demographic

information; Surveys; Incidence

data

Pretest: Knowledge,

Attitude, Behavior;

Posttests: QuantQual;Behavior& PolicyChange;TransferTo othercontexts


Transformative Design Components:Post-Colonial Critical Ethnography

A statement of your evaluation problem or questions (focus). A description of your data-collection methods, including

interviewing, journaling, and coding processes, and how these will be accomplished with the evaluator as a co-performer in the field or participant observer.

An explanation of your ethical methods and how the welfare of the participants will be put first by protecting their rights, interests, privacy, sensibility, and offering reports at key stages to them, including the final report.

A description of the participants in terms of population, geographic location, norms and rules, significant historical and cultural context, and expectations for key informants.

A time frame for entering the field, collecting the data, departing from the field, coding and analysis and completion of the written report, and/or public performance.

Use of a critical theoretical framework in the design, implementation, and dissemination of the study.

Madison (2005)


PAR and Transformative Design

1. The group decides on the focus and questions for the evaluation.

2. Evaluators and participants observe, engage in action, observe and record.

3. Evaluators and participants immerse themselves in action and elaborate and deepen their understandings.

4. Group members reassemble and share their knowledge, using this iteration as an opportunity to revise their plans for the next cycle ofevaluation.

5. This cycle might be repeated between 6 and 10 times depending on the complexity of the evaluation context.

(Heron & Reason, 2006).


Transformative MM and PARGreenwood and Levine (2007) summarize:

“Surveys, statistical analyses, interviews, focus groups, ethnographies, and life histories are all acceptable, if the reason for deploying them has been agreed upon by the AR collaborators and if they are used in a way that does not oppress the participants” (p. 6).


Questions for Thought: Design

What is your reaction to the idea that one group will receive an intervention and another group will not?

Under what circumstances would you accept that format as an ethical course of action?

What other alternatives are possible?(Mertens, 2009, TRE, p. 197)


Gender Analysis and MM

Outcomes, Outputs & ImpactsInputs —> Activities —> Outputs —> Outcomes —

> Impacts Outputs are the products, services, and

capacities that result from the completion of activities Outcomes are the intended or achieved short-term

and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners.

Impacts are the long term effects or change to which the programme, through collective effort with partners, will contribute.

GE/HR evaluation: how did the programme contribute to GE and HR?

http://unifem.org/evaluation_manual/unifem-glossary/outputs/

http://unifem.org/evaluation_manual/unifem-glossary/outcome/

http://unifem.org/evaluation_manual/unifem-glossary/impact/


INDICATORS Indicators are the means by which you

determine progress towards a result or whether an expected result has been achieved.

Developed for ALL levels of results – outputs,

outcomes & impact

Indicators measure (quantitatively or qualitatively) the status of an expected result

15Module 2-1Source: UNIFEM RBM training


INDICATORS Quantitative: focus on numbers & counting

(percentage of women and men in parliament, male and female wage rates, school enrolment for girls and boys)

Qualitative: capture opinions, attitudes and feelings (nature of dissatisfaction, extent of increased awareness)

16Module 2-1


EXAMPLESQuantitative - Can be directly counted & expressed as a

number % of … # of … Frequency of … Ratio of … Amount of … Timeliness of …

Qualitative - Involves perception (can be expressed quantitatively or as narrative)

Level of Satisfaction with … Knowledge of … Ability to … Appropriateness of... Importance of … Use and usefulness of …

17Module 2-1Source: UNIFEM RBM Training.


GENDER SENSITIVE INDICATORS Measures gender-related changes in

society over time

Includes sex-disaggregated indicators that

provide separate measures for women and men

Indicators that are gender-specific to either women or men

18Module 2-1Source: A. Moser. 2007Gender and Indicators


HUMAN RIGHTS INDICATORSAssess both state progress in guaranteeing

human rights and individual programme adherence and promotion of rights.

HR indicators should measure: Improvements in the capacities of rights

holders and duty bearers to realize rights

Improvements in the enjoyments of rights

19Module 2-1

GE & HR RESPONSIVE EVALUATION


Follow-Up and Use:

Dissemination strategies should make evaluation findings accessible and barrier-free to women, including both RHs and DBs Targeting women’s organizations/networks and

knowledge networks User-friendly language Stakeholder workshops that include women and other

groups subject to discrimination (RHs &DBs) Management Response should be issued to ensure

follow-up on key gender and human rights issues


EXAMPLE INDICATORSOutcome indicator: Evidence of changes in

the capability of ministries of xxx to formulate and implement policies responsive to indigenous women

Output indicator: Capacity assessments (focusing on individual competencies within ministries of xxx) indicating increase in relevant knowledge, skills and attitudes within key public institutions

Examples: GE/HR Indicators proportion of target group by sex aware

of the benefits of birth registration, employment to population ratio by age

and sex, gross primary graduation ratio by sex, percentage of women in parliament.

“REALITY CHECK”


Expense and making use of available data; are there too many indicators?

Unrelenting pull to quantitative measures - need a balance of qualitative and quantitative data

Proxy indicators, where measurement takes an more indirect path

Indicators should be disaggregated, as much as possible, by sex, race, ethnicity, age, geographic area

Indicators do not exist in a vacuum and must be tied to a result.

EXERCISE: DEVELOP INDICATORS


Develop two indicators (one qualitative and one quantitative) for the course example.

Sample Relevance Evaluation ?’s How well do the programme objectives

target the identified rights and needs of male and female beneficiaries?

What rights does the programme advance under CEDAW, the Millennium Development Goals and other international development commitments?

http://unifem.org/evaluation_manual/unifem-glossary/goals/

Sample Evaluation Effectiveness ?’s To what extent have the objectives been

achieved, and do the intended and unintended benefits meet fairly the needs of disadvantaged women?

To what extent have the capacities of duty-bearers and rights-holders been strengthened?

Sample Evaluation Efficiency ?’s Could the activities and outputs have been

delivered with fewer resources to the target populations without reducing their quality and quantity?

How has the programme maximized partnerships in the delivery of the programme?

Have UN Woman’s organisational structure, managerial support and coordination mechanisms effectively supported the delivery of the programme?

http://unifem.org/evaluation_manual/unifem-glossary/outputs/

Sample Evaluation Impact ?’s To what extent have efforts been

successful in stopping harmful and discriminatory practices against women?

What is the evidence that the programme enabled the rights-holders to claim their rights more successfully and the duty-holders to perform their duties more efficiently?

Sample Evaluation Sustainability ?’s Is the programme supported by national

and local women’s organisations? Do these organisations demonstrate

leadership commitment and technical capacity to continue to work with the programme or advocate for change?

Additional GE/HR ?’s Equality and non-discrimination: Did the programme benefits

affect equally men and women? Empowerment: Did the budget designate sufficient resources

and level of effort to address the inclusion of disadvantaged or marginalised groups?

Accountability: Were monitoring data (disaggregated according to relevant criteria such as gender, age, ethnicity, location, and income) collected and used to adjust implementation?

Social transformation: Does the intervention’s theory of change include attention to GE and HR?

Participation and inclusion: Did the implementation make systematic and appropriate efforts to include women and men, and/or reach out to disadvantaged groups? UN Women 2011

GE/HR Evaluation Methods Avoid bias: selection of data collection

methods & in sampling methods Potential biases: Gender & Power

(sources able to contribute more easily because of privacy & confidentiality issues ), Class or distance (favoring the more accessible)

Plans for how to include marginalized groups

Mixed methods: Quan & Qual


Rigor applied to methods choices


In the positivist tradition, there are two important tests of knowledge claims:

1. Is the knowledge claim true in this situation? Are the changes observed in the dependent variable due to the effect of the independent variable? (Internal Validity)

2. Is the knowledge true in other situations? Generalizability? (External Validity)


Comparative Studies: Rigor Causal relationship assumed? Competing

explanations? Comparable groups in causal comparative? Third variable cause both predictor and

criterion variables? Sub groups analysis? Correlational: ordering of variables? Predictive studies: Other screening variables?

Level of .8 or better? Reliability and range of variables


Criteria that establish rigor in qualitative methods

Credibility: just like validity in qualitative research – asks if there is a correspondence between the way the respondents actually perceive social constructs and the way the evaluators portray their viewpoints.


(a) persistent observation (b) peer debriefing(c) progressive subjectivity (d) member checks(e) triangulation(f) transferability (g) dependability(h) confirmability(i) authenticity and fairness

Criteria that establish rigor in qualitative methods

TRANSFORMATIVE EVALUATION: Rigor


Emphasizes Human Rights and Social justice Analyses asymmetric power relations Advocates culturally competent relations

between the evaluator and community members

Employs culturally appropriate mixed methods tied to social action

Applies feminist theory, critical race theory, postcolonial and indigenous theories

Mertens (2009) Transformative Research and Evaluation. The Guilford Press.


Authenticity (Transformative criteria)

Fairness: The evaluator presents all value differences, views, and conflicts.

Ontological Authenticity: An individual’s or groups’ conscious experience of the world became more informed and sophisticated.

Catalytic Authenticity: Action was stimulated by the inquiry process.


Power and Privilege How do we understand the dynamics of power when

participatory methods are employed by the powerful? Whose voices are raised and whose are heard? How are these voices mediated as issues of

representation become more complex with the use of participatory methods in larger-scale planning and consultation exercises? (Mertens, 2009, P. 85)

What if I am a member of the community? How does that prepare me to work in that community? What if I am not a member of a community? To what extent is it necessary to share salient characteristics of a community?

How does cultural competency come into the discussion of interactions in evaluation contexts? (Mertens & Wilson, in press)


Criteria for Rigor: Utilization Dissemination Use Management Response Engagement with Stakeholders

Dissemination Purposes Transparency and accountability Informing and improving an organisation’s

work Sharing good practices and ‘how to’ to

advance women’s human rights Sharing lessons with partners on the ground

and building their capacity, and Generating knowledge on how gender

equality, women’s empowerment and human rights can be advanced

Dissemination Language and presentation of the report: use graphs or

pictures, and be written in a manner that is gender and culturally sensitive.

Translation: anticipate the need for translation when planning the evaluation so time and budget can be included)

Making the report public: Within the UN, there is requirement to make all evaluations public (UNEG Norms and Standards) and in UN Women, this is accomplished by having the Evaluation Unit post the evaluation report in the Evaluation Resource Center (which is publicly accessible).

Reaching target audiences in a user-friendly way: lower rates of literacy? The manager needs to be creative in using other dissemination channels, suited to different audiences.

Dissemination Strategies Brochures outlining

key evaluation lessons and recommendations

Annual reports Articles in technical or

organisational newsletters

News releases Press conferences Media appearances

Public meetings, public debates or town halls

Seminars, workshops, and informal group discussions

Electronic media (e-mail, websites, blogs etc.)

Meeting with community leaders, one on one.

Challenges to Use Lack of consensus on recommendations

& required action Those who don’t like the results attack

the process Dissemination is minimal: no funds,

interest or time No follow-up process

Formal Management Response presentation of the response, action or

non action to evaluation recommendations and lessons learned and follow-up or tracking mechanisms.

engage with stakeholders to reflect on the evaluation process, findings, recommendations and lessons learned.

Involve reference group in developing the management response, with the manager.


Agenda for Action How does the information you learned

here apply to the work that you are/will be doing in evaluation?


Resources Mertens, D. M. & Wilson, A. (in

press). Program Evaluation. NY: Guilford.

Mertens, D. M. (2010). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 3rd ed. Thousand Oaks, CA: Sage.

Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford

Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage.


Contact information Donna M. Mertens, Gallaudet University

[email protected]

mailto:[email protected]

Documents

UNWOMEN/IPEN Transformative Mixed Methods Evaluation: Day 3 Mixed Methods