What is evaluation What is evaluation knowledge for practice?knowledge for practice?

Thomas A. SchwandtThomas A. Schwandt

22

Evaluation as EvidenceEvaluation as Evidence--forfor--UseUse

Reinvigorated link between Reinvigorated link between evaluation and scientific thinking evaluation and scientific thinking (e.g., experimentalism, realist (e.g., experimentalism, realist evaluation, theoryevaluation, theory--based evaluation)based evaluation)Collections of evidence modeled on Collections of evidence modeled on Cochran CollaborationCochran CollaborationDefining evaluation in the context of Defining evaluation in the context of managerialismmanagerialism

33

OutlineOutline

The meaning of The meaning of evaluationevaluation and and practicepracticeHow they can be relatedHow they can be relatedSignificance of this analysis for the Significance of this analysis for the way we think about evaluationway we think about evaluation

44

Why approach the topic this Why approach the topic this way?way?

What are we assuming about ‘evaluation’and ‘practice’ when we direct our evaluative lens on the work of practitioners?

Learning from the ‘practice turn’

The importance of language

55

What is What is ““evaluationevaluation””? ? Two ways to talk about evaluation:Two ways to talk about evaluation:•• E E valuationvaluation•• e e valuation valuation Both concerned with making distinctions Both concerned with making distinctions

of worthof worth——that is, decisions about the that is, decisions about the value, merit, significance of some action value, merit, significance of some action taken or about to be takentaken or about to be taken

Each reflects a different rationality & a Each reflects a different rationality & a different idea of evaluation knowledgedifferent idea of evaluation knowledge

66

E E valuationvaluationThe most familiar to usThe most familiar to usTools and approaches for making Tools and approaches for making decisions about value guided by a general decisions about value guided by a general logic:logic:•• Establish criteria of meritEstablish criteria of merit•• Generate evidence of performance on criteriaGenerate evidence of performance on criteria•• Synthesize performance into overall judgmentSynthesize performance into overall judgment

Professional activityProfessional activity

77

Examples of Examples of E E valuationvaluation

Functional outcome assessmentFunctional outcome assessmentQuality of care approachQuality of care approach——performance monitoring and auditingperformance monitoring and auditingEvidenceEvidence--based practicebased practice

88

Functional AssessmentFunctional Assessment

Functional Assessment Taxonomy for Functional Assessment Taxonomy for disability determinationdisability determination•• Operationally defined and measured Operationally defined and measured

constructs for workconstructs for work--related assessments related assessments used preused pre-- and postand post--interventionintervention

99

Quality of Care Approach:Quality of Care Approach:Evaluating the service systemEvaluating the service system

DomainDomainTechnical careTechnical care

Process of careProcess of care

Structure of careStructure of care

DefinitionDefinitionClient outcomesClient outcomes

Organization of service Organization of service team, activities directed at team, activities directed at client motivation, etc.client motivation, etc.

Qualities of space, Qualities of space, facilities, equipment, etc.facilities, equipment, etc.

1010

EvidenceEvidence--based practice based practice approachapproach

What worksWhat works----causal efficacy causal efficacy of the treatment/interventionof the treatment/intervention

1111

EvidenceEvidence--based practice based practice approachapproach

Taxonomy of evidenceTaxonomy of evidenceLevel I Level I –– Evidence from Evidence from RCTsRCTs or systematic reviewsor systematic reviewsLevel II Level II –– Evidence of one RCTEvidence of one RCTLevel III Level III –– Evidence from cohort or caseEvidence from cohort or case--controlled controlled

studiesstudiesLevel IV Level IV –– Evidence from nonEvidence from non--experimental studiesexperimental studiesLevel V Level V –– Opinions of authorities, clinical Opinions of authorities, clinical

experience, descriptive studiesexperience, descriptive studies

1212

What What E E valuation Assumes About valuation Assumes About Practice, Rationality, & KnowledgePractice, Rationality, & Knowledge

1)1) World of actors with intentional World of actors with intentional states/ individual motivations states/ individual motivations distinct from and acting ondistinct from and acting on

2)2) A preA pre--existing world of existing world of ‘‘practicepractice’’. . Thus,Thus,

3)3) Assume relationship of detachmentAssume relationship of detachment

1313

Individual mind of actor/ Individual mind of actor/ External world of actionsExternal world of actions

1414

What What E E valuation Assumes About valuation Assumes About PracticePractice

4)4) Practice is something deliberately Practice is something deliberately chosen and aligned with strategies chosen and aligned with strategies and goals in the minds of actorsand goals in the minds of actors

5)5) This practice is capable of This practice is capable of observation and classification by observation and classification by the evaluator in terms of inputs, the evaluator in terms of inputs, processes, and outputsprocesses, and outputs

1515

What What EEvaluation Assumes About valuation Assumes About PracticePractice

6)6) The authority of evaluation resides The authority of evaluation resides in its ability to supersede everyday in its ability to supersede everyday experience in the way in which it experience in the way in which it studies practicestudies practice

1616

E E valuation authorityvaluation authority

Performance of A Practice

1717

What What E E valuation Assumes About valuation Assumes About PracticePractice

7)7) Practice = an instrumental means Practice = an instrumental means to solve problems; governed by to solve problems; governed by instrumental rationalityinstrumental rationality

8)8) Practice = complex set of Practice = complex set of techniques, attitudes, dispositionstechniques, attitudes, dispositions

9)9) Normative state of affairs for Normative state of affairs for practice = clarity, relative certainty, practice = clarity, relative certainty, orderorder

1818Lack of clarity and confusion and ambiguity in practice….

1919

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East OfficeWest OfficeNorth Office

BRINGING ORDER TO PRACTICE

2020

Ah! Clarity and the elimination of ambiguity….!

2121

What What E E valuation Assumes About valuation Assumes About PracticePractice

‘Outside’, Expert Knowledge

Problems ofAction inPractice

Applied

to

2222

What What E E valuation Assumes About valuation Assumes About PracticePractice

Because practices are local, Because practices are local, contingent, contextual, contingent, contextual, E E valuation valuation knowledge must be adapted to the knowledge must be adapted to the demands of a given contextdemands of a given contextThe problem, however, lies not in the The problem, however, lies not in the nature of the knowledge but in the nature of the knowledge but in the ‘‘contextcontext’’

2323

SummarySummary

Actors are selfActors are self--conscious, intentionalconscious, intentionalActions are guided by clear goals and Actions are guided by clear goals and strategies; they are purposeful and strategies; they are purposeful and plannedplannedAction can be reconstructed in logic Action can be reconstructed in logic of explanationof explanationAll of this is capable of observation All of this is capable of observation and classification by the evaluatorand classification by the evaluator

2424

SummarySummary

A nonA non--collaborative collaborative ‘‘sciencescience’’ of of evaluationevaluation““SuccessSuccess”” of the evaluation judged of the evaluation judged by independent standards for by independent standards for rational behavior on the part of rational behavior on the part of evaluation expertsevaluation experts

2525

ee valuationvaluation

A kind of evaluation A kind of evaluation knowledge and rationality in knowledge and rationality in agreement with everyday lifeagreement with everyday life

2626

What What e e valuation assumes about valuation assumes about practicepractice

Practice is not simply what people Practice is not simply what people ‘‘dodo’’Practice is a social site where Practice is a social site where identities of practitioners and identities of practitioners and strategies are strategies are relationallyrelationally coco--constructedconstructed

2727

““DwellingDwelling””

A mode of engagement with the A mode of engagement with the world (of people and objects) in world (of people and objects) in which the familiar things around us which the familiar things around us are nonare non--deliberately and effortlessly deliberately and effortlessly coco--opted into our current activity to opted into our current activity to form an extension of our being and form an extension of our being and doing doing

2828

RelationalityRelationalityThe ceaseless actions of daily life constitute a fluctuating network of connections and disconnections with the various objects that extend and reflect our sense of human agency. Instead of an external world of stable objects that support our needs and purposes, the space and time of daily life begins to look like a restless scene of acts in permanent suspension… Acts…connect and disconnect. They express relationships between things rather than the things themselves. (Cooper, 2005, p. 1690-1691)

2929

RelationalityRelationality

I have in mind the hundreds of practical judgments, the everyday taken-for-granted routines and practices, the explicit and tacit knowledge that is brought to bear on concrete situations, the moving about in the legal-moral environment of large administrative bureaucracies, the mastering of difficult human-emotional situations, the negotiating of discretionary space, and the interactive give and take with colleagues that make up everyday public administration. (Wagenaar, 2004, p.644)

3030

““PracticePractice””

Is inherently dialogical & interactiveIs inherently dialogical & interactiveIt is about transactions with others It is about transactions with others that are simultaneously cognitive and that are simultaneously cognitive and emotiveemotiveIs fluid, changeable, dynamicIs fluid, changeable, dynamicIs about the particularIs about the particularIs indeterminate (impossible to know Is indeterminate (impossible to know precisely)precisely)

3131

““PracticePractice””

Requires a kind of knowledge we call Requires a kind of knowledge we call practical judgment, wise judgment, practical judgment, wise judgment, practical wisdom, practical wisdom, phronesisphronesis, that, in , that, in turn, requires perception and turn, requires perception and deliberation or judgmentdeliberation or judgment

3232

Practical perceptionPractical perception[The] capacity to identify with clarity and discernment the features of a complex situation that are significant for determining the most appropriate action. It brings to the fore aspects of a situation that have gone unnoticed and gives one an intuitive sense of what is important for making a moral choice. Practical perception is cultivated through one’s experiences of moral choice and through observation of the choices made by [others]. (Polkinghorne, 2004, p. 117)

3333

What does this view of practice What does this view of practice have to do with the way we think have to do with the way we think

about evaluation?about evaluation?

A different rationality and A different rationality and a different kind of knowledge a different kind of knowledge

3434

Why Why ee valuation?valuation?

It draws our attention to the kind of It draws our attention to the kind of decisions about distinctions of worth decisions about distinctions of worth inherent in our practicesinherent in our practicesIt is the practical art of living well, of It is the practical art of living well, of realizing the realizing the ‘‘goodgood’’ constitutive of a constitutive of a morally worthwhile form of human morally worthwhile form of human activity (e.g., teaching counseling, activity (e.g., teaching counseling, parenting, social work, etc.)parenting, social work, etc.)

3535

ee valuationvaluation

Focused on the moralFocused on the moral--political political activity called praxisactivity called praxisAntidote to the view that the only Antidote to the view that the only rational rational ‘‘sciencescience’’ of evaluation is a of evaluation is a methodical sciencemethodical science

3636

ee valuationvaluation

A science of practical philosophy; a A science of practical philosophy; a science that guides praxisscience that guides praxisA science that reclaims the sphere of A science that reclaims the sphere of praxis from its modern assimilation praxis from its modern assimilation into the sphere of technical and into the sphere of technical and instrumental rationalityinstrumental rationality

3737

What is What is praxispraxis??1.1. Making wise choices as we pursue Making wise choices as we pursue

common goalscommon goals2.2. Acting rationally in the absence of rules Acting rationally in the absence of rules

and proceduresand procedures3.3. Arriving at understandings of what it is Arriving at understandings of what it is

right to do through conversationright to do through conversation4.4. Allowing ourselves to be questionedAllowing ourselves to be questioned5.5. Recognizing that our decisions are always Recognizing that our decisions are always

imperfect, provisional, never imperfect, provisional, never ‘‘finishedfinished’’

3838

A practical science of A practical science of ee valuationvaluation

““Practical knowledge and Practical knowledge and understanding can only be developed understanding can only be developed and advanced by practitioners and advanced by practitioners engaging in the kind of dialogue and engaging in the kind of dialogue and conversation through whichconversation through which…… [the] [the] assumptions implicit in their practice assumptions implicit in their practice can be made explicit and their can be made explicit and their collective understanding of their collective understanding of their praxis transformedpraxis transformed”” (Carr, 2006)(Carr, 2006)

3939

A practical science of A practical science of ee valuationvaluation

Goal is not knowledge to control or Goal is not knowledge to control or repair what practitioners dorepair what practitioners doGoal is to initiate and sustain Goal is to initiate and sustain processes of critical reflectionprocesses of critical reflectionEvaluator as collaboratorEvaluator as collaboratoree valuation is dialogical and socially valuation is dialogical and socially cooperativecooperative

4040

Socially cooperative form of Socially cooperative form of inquiryinquiry……

4141

Disagreement as productive Disagreement as productive feature of feature of praxispraxis……

4242

Critical reflection on Critical reflection on praxispraxis

Mapping a practiceMapping a practice----puzzles, puzzles, dilemmas, challenges; dilemmas, challenges; communicative strategies, communicative strategies, dispositions, normative expectations, dispositions, normative expectations, guiding concepts, rules, proceduresguiding concepts, rules, procedures

4343

Critical reflection on Critical reflection on praxispraxis

Examining practical knowledge:Examining practical knowledge:1.1. Rules of practiceRules of practice2.2. Practical principlesPractical principles3.3. ImagesImages----Embodied, experiential Embodied, experiential

complexes of meaning that guide complexes of meaning that guide decisionsdecisions

4444

Final thoughtsFinal thoughts……

EEvaluation and valuation and eevaluation valuation ≠≠ Macro Macro versus micro view of evaluationversus micro view of evaluationThe important issue is that each is a The important issue is that each is a different form of rationality and different form of rationality and requires a different form of requires a different form of knowledgeknowledge

4545

Different orientations to Different orientations to everyday lifeeveryday life

Instrumental and Instrumental and technical rationalitytechnical rationalityPractice is an Practice is an objectobject--like entity like entity that can be that can be inspectedinspectedEvaluation Evaluation knowledge is knowledge is means to an endmeans to an end

Practical rationalityPractical rationalityPractice is a Practice is a relational dwellingrelational dwellingEvaluation Evaluation knowledge is knowledge is knowledge about knowledge about how to engage in how to engage in critical reflection critical reflection on praxison praxis

4646

What is What is ‘‘evaluationevaluation’’ knowledge knowledge for?for?

KnowledgeKnowledge--forfor--action: the action: the application of application of knowledge and knowledge and technique to the technique to the instrumental instrumental problems of problems of practice (e.g., practice (e.g., effectiveness and effectiveness and efficiency)efficiency)

KnowledgeKnowledge--inin--action: action: Understanding how Understanding how to make wise to make wise decisions acquired decisions acquired from critical from critical reflection on reflection on experience of experience of actual situations; actual situations; means and ends means and ends defined defined interactivelyinteractively

4747

Consequences of Consequences of ‘‘EE valuationvaluation’’ knowledgeknowledge

Dominant belief that everyday life Dominant belief that everyday life should be rationalizedshould be rationalized----reordered reordered according to a technical model of according to a technical model of applied knowledgeapplied knowledgeTendency to surrender responsibility Tendency to surrender responsibility for decisions about the value of for decisions about the value of practice to expertspractice to experts

4848

Consequences of Consequences of ‘‘ee valuationvaluation’’ knowledgeknowledge

Recognize that practitioners are not Recognize that practitioners are not functionariesfunctionaries----simply carrying out specialized simply carrying out specialized tasks on behalf of societytasks on behalf of societyRecognize that practices are not simply Recognize that practices are not simply ‘‘delivery delivery mechanismsmechanisms’’ for providing services to clientsfor providing services to clientsRecognize that it is in practices such as social Recognize that it is in practices such as social work, teaching, etc. that we become aware of work, teaching, etc. that we become aware of what it means to be human, to live together, to what it means to be human, to live together, to flourishflourish

4949

Consequences of Consequences of ‘‘ee valuationvaluation’’ knowledgeknowledge

Recognize that praxis is not simply a matter of Recognize that praxis is not simply a matter of finding suitable means to achieve finding suitable means to achieve pregivenpregivenpurposes (this is technology)purposes (this is technology)Recognize that praxis is matter of collectively Recognize that praxis is matter of collectively making wise choices in pursuit of common goalsmaking wise choices in pursuit of common goalsRecognize that we are morally and politically Recognize that we are morally and politically responsible for our choicesresponsible for our choices

5050

EVALUATION SHAPES OUR EVALUATION SHAPES OUR SENSE OF SELFSENSE OF SELF

5151

At the heart of practices are At the heart of practices are practitioners wrestling with making practitioners wrestling with making countless decisions about what is the countless decisions about what is the right, honest, just, and wise thing to do right, honest, just, and wise thing to do when facing a client, student, patient, or when facing a client, student, patient, or subordinate.subordinate.

If we want better practices, programs, If we want better practices, programs, and policies, then we need to help them and policies, then we need to help them do this kind of do this kind of e e valuation.valuation.

5252

Thank youThank youfor listeningfor listening