Social Justice Amidst Standards and Accountability Donna M. Mertens, Keynote Independent Consultant...
If you can't read please download the document
Social Justice Amidst Standards and Accountability Donna M. Mertens, Keynote Independent Consultant Minnesota Evaluation Studies Institute, MESI Spring
Social Justice Amidst Standards and Accountability Donna M.
Mertens, Keynote Independent Consultant Minnesota Evaluation
Studies Institute, MESI Spring Training Minneapolis MN March
2015
Slide 2
Why this topic? Why now? Social Justice Amidst Standards and
Accountability 3/11/2015 Mertens Keynote MN MESI March 2015 2
Slide 3
2015 International Year of Evaluation EvalPartners, UNICEF, UN
Women, and the International Organization for Cooperation in
Evaluation Peer 2 Peer Initiative gender and equity focused
evaluation 3/11/2015 Mertens Keynote MN MESI March 2015 3
Slide 4
Need for Social Justice Link to Evaluation: Pervasiveness of
racism and other isms 3/11/2015 Mertens Keynote MN MESI March 2015
4
Slide 5
Need for a Social Justice Lens Marie Battiste (2000): society
is sorely in need of what Aboriginal knowledge has to offer Chilisa
(2005) it is an issue of life and death Brookes (2006) Failure to
include racism as a potential contributor to disparities 3/11/2015
Mertens Keynote MN MESI March 2015 5
Slide 6
My hypothesis: If we begin by prioritizing social justice and
human rights And we appropriately involve community members in the
evaluation process, Then we will increase the probability of social
transformation as a result of our evaluation. 3/11/2015 Mertens
Keynote MN MESI March 2015 6
Slide 7
What does our evidence lack in credibility from a social
justice perspective? 3/11/2015 Mertens Keynote MN MESI March 2015
7
Slide 8
Social Justice Standards Accountability 3/11/2015 Mertens
Keynote MN MESI March 2015 8
Slide 9
Accountability Question Did it work? Should we keep spending
our money on it? 3/11/2015 Mertens Keynote MN MESI March 2015
9
Slide 10
Social Justice Questions Was it the right thing? Was it chosen
and/or developed and implemented in culturally responsive ways?
Were contextual issues of culture, race/ethnicity, gender,
disability, deafness, religion, language, immigrant or refugee
status, age or other dimensions of diversity used as a basis for
discrimination and oppression addressed? How were issues of power
addressed? Do we want to continue to spend money on things that
dont work? 3/11/2015 Mertens Keynote MN MESI March 2015 10
Slide 11
Resources for Evaluators AEA Guiding Principles AEA Public
Statement on Cultural Competency Joint Committee Program Evaluation
Standards Evaluators who have dedicated their lives towards
furthering social justice Communities who want social justice
3/11/2015 Mertens Keynote MN MESI March 2015 11
Slide 12
Good news; bad news for social justice in evaluation 1.2%
(n=22) in Harners (2014) survey of evaluators identified their
theoretical frame as Social Justice 69% (n=819) of 1,187 evaluators
either strongly or somewhat agreed with this statement: Evaluation
should focus on bringing about social justice. 3/11/2015 Mertens
Keynote MN MESI March 2015 12
Slide 13
AEA Guiding Principles (2004): Systematic Inquiry, Competence,
Integrity/Honesty, Respect for People, and Responsibilities for
General and Public Welfare To ensure recognition, accurate
interpretation and respect for diversity, evaluators should ensure
that the members of the evaluation team collectively demonstrate
cultural competence. Cultural competence would be reflected in
evaluators seeking awareness of their own culturally-based
assumptions, their understanding of the worldviews of
culturally-different participants and stakeholders in the
evaluation, and the use of appropriate evaluation strategies and
skills in working with culturally different groups. Diversity may
be in terms of race, ethnicity, gender, religion, socio-economics,
or other factors pertinent to the evaluation context. (Competence
B2) 3/11/2015 Mertens Keynote MN MESI March 2015 13
Slide 14
AEA Guiding Principle: Cultural Competence Linkage with
Transformative Axiology and Epistemology To ensure recognition,
accurate interpretation, and respect for diversity, evaluators
should ensure that the members of the evaluation team collectively
demonstrate cultural competence (American Evaluation Association,
Public Statement on Cultural Competence in Evaluation, 2011)
3/11/2015 Mertens Keynote MN MESI March 2015 14
Slide 15
What is cultural competence? Cultural competence is not a state
at which one arrives; rather, it is a process of learning,
unlearning, and relearning. It is a sensibility cultivated
throughout a lifetime. Cultural competence requires awareness of
self, reflection on ones own cultural position, awareness of others
positions, and the ability to interact genuinely and respectfully
with others. Culturally competent evaluators refrain from assuming
they fully understand the perspectives of stakeholders whose
backgrounds differ from their own. Cultural competence is context
dependent. (AEA 2011) 3/11/2015 Mertens Keynote MN MESI March 2015
15
Slide 16
Evaluation and Validity and Cultural Competence accurately and
respectfully reflect the life experiences and perspectives of
program participants in their evaluations. establish relationships
that support trustworthy communication among all participants in
the evaluation process. draw upon culturally relevant, and in some
cases culturally specific, theory in the design of the evaluation
and the interpretation of findings. select and implement design
options and measurement strategies in ways that are compatible with
the cultural context of the study. (AEA 2004, p. 6) 3/11/2015
Mertens Keynote MN MESI March 2015 16
Slide 17
Recognize the dynamics of power Cultural groupings are ascribed
differential status and power, with some holding privilege that
they may not be aware of and some being relegated to the status of
other. For example, language dialect and accent can be used to
determine the status, privilege, and access to resources of groups.
(AEA, 2004, p. 7) 3/11/2015 Mertens Keynote MN MESI March 2015
17
Slide 18
Standards for Good Evaluation (Yarborough et al. 2011) Utility:
evaluator credibility, attention to stakeholders, negotiated
purposes, explicit purposes Feasibility: management, practical,
contextual viability (recognize, monitor, and balance the cultural
and political interests and needs of individuals and groups)
Propriety: responsive and inclusive, formal agreements, human
rights and respect, clarity and fairness, transparency and
disclosure. Accuracy: valid, reliable, explicit program and context
descriptions, sound designs and analysis Evaluation accountability:
meta-evaluation 3/11/2015 Mertens Keynote MN MESI March 2015
18
Slide 19
Accountability and Social Justice Code word for RCT?
Accountable to whom? Ways to strengthen arguments about
accountability are not limited to RCTs Theoretical streams in
evaluation provide options for demonstrating accountability
3/11/2015 Mertens Keynote MN MESI March 2015 19
Slide 20
Evaluation Theory and Social Justice Alkins Tree: Methods, Use
& Values Branches Mertens Keynote MN MESI March 201520
ValuesMethods Use 3/11/2015
Slide 21
Mertens & Wilson 2012: Methods, Use, Values, & Social
Justice Branches Mertens Keynote MN MESI March 201521 Methods Use
Values Social Justice 3/11/2015
Slide 22
Metaphor: Tree or Water? Mertens Keynote MN MESI March 2015 22
3/11/2015
Slide 23
Paradigms & Branches ParadigmBranch Post-positivistMethods
ConstructivistValues TransformativeSocial Justice PragmaticUse
Mertens Keynote MN MESI March 2015 23 3/11/2015
Slide 24
Transformative Theories Feminist theories Critical Race
Theories Critical Theories Human Rights Theories Disability Rights
Theories Transformative Participatory Action theories Indigenous
theories Deafness rights theories Queer theories 3/11/2015 Mertens
Keynote MN MESI March 2015 24
Slide 25
Transformative Paradigm PhilosophicalAssumptions
AxiologyRespect for cultural norms; support for human rights and
social justice; reciprocity OntologyIssues of power & critical
interrogation of multiple realities: social, political, cultural,
economic, race/ethnic, gender, age, religion and disability values
to unmask those that sustain an oppressive status quo
EpistemologyIssues of power & Interactive link; knowledge is
socially and historically located; trusting relationship.
Methodology Qualitative (dialogic)/ Quantitative / Mixed Methods;
Context Mertens Keynote MN MESI March 2015 25 3/11/2015
Slide 26
Court Access Project Start from community Advisory board formed
Begin understandings from the vantage point of the least privileged
Work with an awareness of diversity in the community Provide
respectful support for engagement Use a cyclical approach to data
collection and use 3/11/2015 Mertens Keynote MN MESI March 2015
26
Slide 27
Example: HIV/AIDS Prevention in Botswana Botswana youth:
addressing power inequities in the fight against HIV/AIDS using a
transformative lens Mertens Keynote MN MESI March 2015 27
3/11/2015
Slide 28
Transformative Cyclical Mixed Methods Design Mertens Keynote MN
MESI March 2015 28 Stage 4 Sequential Stage 5 Concurrent Stage 3
Sequential Stage 2 Concurrent Stage 1 Qual Assemble team; read
documents; engage in dialogues Preliminary studies: youth, older
men Process eval Pilot intervention: Observations, Interviews,
Surveys Demographic information; Surveys; Incidence data Pretest:
Knowledge, Attitude, Behavior; Post tests: Quant Qual; Behavior
& Policy Change; Transfer To other contexts RCT 3/11/2015
Slide 29
Indigenous Contributions Social justice as a frame for
evaluation Engagement with indigenous communities 3/11/2015 Mertens
Keynote MN MESI March 2015 29
Slide 30
Indigenous Paradigm Chilisa (2012) Wilson (2008) Linda T. Smith
(1999) Marie Battiste (2000) LaFrance & Crazy Horse (2009)
Relational Axiology Relational Ontology Relational Epistemology
Relational Methodology (Chilisa, 2012) 3/11/2015 Mertens Keynote MN
MESI March 2015 30
Slide 31
Ethics and Indigeneity Connections between people, past,
present, and future, all living and nonliving things (ubuntu)
Respect and reciprocity: listens, pays attention, acknowledged, and
crates space for the voices and knowledge systems of Indigenous
people Contribute to a better future Spirituality 3/11/2015 Mertens
Keynote MN MESI March 2015 31
Slide 32
Questions What is credible evidence? What criteria establish
credibility of evidence? What does our evidence lack in
credibility? What sources give us insight into credibility? What is
the place of social justice in credibility? How do voices of
marginalized communities enhance our understandings of credibility?
Why is it important for researchers to understand marginalized
communities perspectives of social justice? Mertens Keynote MN MESI
March 2015 3/11/2015 32
Slide 33
Methodological Questions What data collection methods can the
evaluator use to become better acquainted with the various
stakeholder groups? How can the evaluator design the study so that
the community members are included in decision making? How could
the use of mixed methods contribute to the quality of the study?
How can the evaluator design the methods to increase the
probability of furthering social justice? 3/11/2015 Mertens Keynote
MN MESI March 2015 33
Slide 34
Water as Metaphor Eventually, all things merge into one, and a
river runs through it. The river was cut by the world's great flood
and runs over rocks from the basement of time. - Norman Fitzroy
Maclean When you put your hand in a flowing stream, you touch the
last that has gone before and the first of what is still to come. -
Leonardo da Vinci My soul is full of longing For the secret of the
Sea, And the heart of the great ocean Sends a thrilling pulse
through me Henry Wadsworth Longfellow
http://www.finestquotes.com/select_quote-category-Water-page-1.htm#ixzz2968zeXR9
Mertens Keynote MN MESI March 2015 34 3/11/2015
Slide 35
Resources Mertens, D. M. & Wilson, A. (2012). Program
Evaluation Theory and Practice: A Comprehensive Guide. NY:
Guilford. Mertens, D. M. (2015). Research and evaluation in
education and psychology: Integrating diversity with qual, quant
and mixed methods. 4th ed. Thousand Oaks, CA: Sage. Mertens, D. M.
(2009). Transformative research & evaluation. NY: Guilford.
Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social
Research Ethics. Thousand Oaks, CA: Sage. American Evaluation
Association (2011). Public Statement on Cultural Competence in
Evaluation. AEA. Mertens Keynote MN MESI March 2015 35
3/11/2015
Slide 36
Contact information Donna M. Mertens, PhD Independent
Consultant [email protected] Mertens Keynote MN MESI
March 2015 36 3/11/2015