Qualitative Assessment Qualitative Assessment

““Qualitative inquiry cultivates the most useful of Qualitative inquiry cultivates the most useful of all human capacities—the capacity to learn from all human capacities—the capacity to learn from others.”others.”Halcom’s Evaluation Laws.Halcom’s Evaluation Laws. Cited in Patton (1990). Cited in Patton (1990). Qualitative Evaluation and Research MethodsQualitative Evaluation and Research Methods. pp. 7.. pp. 7.

Moving Forward with AssessmentMoving Forward with AssessmentDifficult Dialogues InitiativeDifficult Dialogues Initiative

Assessment ConferenceAssessment ConferenceFord Foundation, NY, NY Ford Foundation, NY, NY

October 16-17, 2008October 16-17, 2008(H. Hern(H. Hernáández-Gravelle)ndez-Gravelle)

In addition to works cited, the author is indebted to the work of M. Patton; A. Driscoll; B.A. Miller; D.T. Spaulding; L.F. Gardiner, C. Anderson, and B.L. Cambridge; T.A. Angelo and K.P. Cross; and C. McTighe Musil, M. Garcia, Y.T. Moses, and D.G. Smith.

Qualitative AssessmentQualitative Assessment Two Definitions Two Definitions

““An inquiry process of understanding a social or An inquiry process of understanding a social or human problem, based on building a complex, human problem, based on building a complex, holistic picture, formed with words, reporting holistic picture, formed with words, reporting detailed views of informants, and conducted in a detailed views of informants, and conducted in a natural setting.”natural setting.”Creswell (1994). Creswell (1994). Research design: Qualitative & Research design: Qualitative & quantitative approaches.quantitative approaches. Cited in Detlor (2004). Cited in Detlor (2004). Towards Towards Knowledge Portals.Knowledge Portals.

““Explores and tries to understand people's Explores and tries to understand people's beliefs, experiences, attitudes, behaviors and beliefs, experiences, attitudes, behaviors and interactions. It generates non-numerical data…”interactions. It generates non-numerical data…”http://new.wales.gov.uk/about/aboutresearch/social/http://new.wales.gov.uk/about/aboutresearch/social/glossary/?lang=englossary/?lang=en..

Qualitative AssessmentQualitative Assessment TidbitsTidbits 1899: W.E. Dubois, undertook first social survey in the 1899: W.E. Dubois, undertook first social survey in the

United States.United States.

1900’s: Early phase of qualitative research. Rooted in Social 1900’s: Early phase of qualitative research. Rooted in Social Investigation.Investigation.

1950’s: Government provided grants in support of 1950’s: Government provided grants in support of educational research. Preferred modality is qualitative educational research. Preferred modality is qualitative research.research.

Late 1960’s: Qualitative research came into being. Late 1960’s: Qualitative research came into being. Development of socially responsive programs. Development of socially responsive programs.

Late 1900’s: Accountability movement in education results Late 1900’s: Accountability movement in education results in increasing use of qualitative research and assessment.in increasing use of qualitative research and assessment.

Why DDI & Qualitative Assessment?Why DDI & Qualitative Assessment?

Well suited in areas that are complex, multidimensional Well suited in areas that are complex, multidimensional sensitive and nuanced.sensitive and nuanced.Illustrative DDI Project Titles-Illustrative DDI Project Titles-

Engaging Controversy: Religion, freedom and the topics of identityEngaging Controversy: Religion, freedom and the topics of identity

Imagining the Future: Dissent, dialogue and the freedom to inquireImagining the Future: Dissent, dialogue and the freedom to inquire

Practicing Pluralism: Interactive theater, campus climate and academic freedomPracticing Pluralism: Interactive theater, campus climate and academic freedom

The open nature of qualitative approach allows the The open nature of qualitative approach allows the subjects to respond according to their own framework.subjects to respond according to their own framework.““In assessing the impact of a faculty development workshop in adoption of new In assessing the impact of a faculty development workshop in adoption of new pedagogy, the teachers’ experience, reasons for participating in the workshop or the pedagogy, the teachers’ experience, reasons for participating in the workshop or the students that she teaches may impact her view of the value of the workshop and her students that she teaches may impact her view of the value of the workshop and her sense of its applicability.”sense of its applicability.”

Quote from grantee.Quote from grantee.

Helps to understand a program / situation as a whole.Helps to understand a program / situation as a whole.““..the holistic approach assumes that the whole is understood as complex system that is ..the holistic approach assumes that the whole is understood as complex system that is greater than the sum of its parts.”greater than the sum of its parts.”Patton (1990). Patton (1990). Qualitative Evaluation and Research MethodsQualitative Evaluation and Research Methods..

Why DDI & Qualitative Assessment?Why DDI & Qualitative Assessment?ContinuedContinued

To provide a richer, more descriptive picture of the To provide a richer, more descriptive picture of the information being gathered or responses to the information being gathered or responses to the questions being asked. For example, assessing and questions being asked. For example, assessing and describing in a rich manner the quality of increased describing in a rich manner the quality of increased capacity for exchange between Arab and Israeli capacity for exchange between Arab and Israeli studentsstudents..

On college campuses addressing pluralism issues, On college campuses addressing pluralism issues, qualitative assessment can in itself serve as a qualitative assessment can in itself serve as a process of inclusion and appreciation for multiple process of inclusion and appreciation for multiple values and views.values and views.

SOME CHARACTERISTICS OF THE SOME CHARACTERISTICS OF THE QUALITATIVE PROCESSQUALITATIVE PROCESS

Process extends through data reporting, data analysis, and Process extends through data reporting, data analysis, and utilization of findings.utilization of findings.

Researcher’s lens can impact data gathering and interpretation. Researcher’s lens can impact data gathering and interpretation. Expertise in both assessment and subject area are significant.Expertise in both assessment and subject area are significant.

It is complex and requires multiple iterations to better uncover It is complex and requires multiple iterations to better uncover information (experiences, perspectives, views).information (experiences, perspectives, views).

Large volumes of data can accumulate quickly. Allocation of time Large volumes of data can accumulate quickly. Allocation of time and resources for data collection and analysis is imperative.and resources for data collection and analysis is imperative.

Value: You get out what you put into it!Value: You get out what you put into it!

Qualitative Approaches Used Qualitative Approaches Used by DDIby DDI

FOCUS GROUPS: In qualitative assessment you go to the FOCUS GROUPS: In qualitative assessment you go to the source (naturalistic group)source (naturalistic group)

INTERVIEWS: direct sourceINTERVIEWS: direct source

PARTICIPANT OBSERVATIONS: To fully describe the process, PARTICIPANT OBSERVATIONS: To fully describe the process, you want to observe ityou want to observe it

PORTFOLIO ASSESSMENT: inductivePORTFOLIO ASSESSMENT: inductive

FILM DOCUMENTATION: profiles and picturesFILM DOCUMENTATION: profiles and pictures

Assessment at its CoreAssessment at its Coreis aboutis about

Student Learning OutcomesStudent Learning Outcomes Question: How do we get from here to Question: How do we get from here to

there?there?• Pluralism is ongoing work Pluralism is ongoing work • Assessment is an ongoing processAssessment is an ongoing process

Tools to assess student learning Tools to assess student learning outcomes:outcomes: • PortfolioPortfolio• RubricsRubrics• In classroom exercise (presentations, In classroom exercise (presentations,

debate)debate)• JournalJournal

ENHANCING DDI INDICATORS OF ENHANCING DDI INDICATORS OF SUCCESS IN QUALITATIVE ASSESSMENTSUCCESS IN QUALITATIVE ASSESSMENT

Ensuring indicators of success are sound:Ensuring indicators of success are sound: Important Important particularly for subjects and areas that are considered outside of particularly for subjects and areas that are considered outside of the core curriculum or where the topics are controversial:the core curriculum or where the topics are controversial:““Unearth narratives about the complex history of race”Unearth narratives about the complex history of race”Quote from grantee.Quote from grantee.

Members check-in:Members check-in: This process can also serve to inform the This process can also serve to inform the constituencies and to begin process of change.constituencies and to begin process of change.

TriangulationTriangulation: : Using multiple sources to confirm findings.Using multiple sources to confirm findings.

Enlisting assessment advice and support from experienced Enlisting assessment advice and support from experienced and knowledgeable colleagues: and knowledgeable colleagues: Serves as a form of Serves as a form of triangulation and check-in. Can address gaps in assessment triangulation and check-in. Can address gaps in assessment capacity.capacity.

Detailed reporting:Detailed reporting: Necessary to illustrate how the assessment Necessary to illustrate how the assessment team arrived at findings and to project the richness of the data.team arrived at findings and to project the richness of the data.

DISTINGUISH ANECDOTAL EVIDENCE DISTINGUISH ANECDOTAL EVIDENCE FROM QUALITATIVE DATAFROM QUALITATIVE DATA

Unsolicited responses to programs in the form of criticism or Unsolicited responses to programs in the form of criticism or praise is considered anecdotal evidence. Such evidence is praise is considered anecdotal evidence. Such evidence is important and worthy of consideration, but it does not important and worthy of consideration, but it does not constitute formal assessment. Qualitative approaches are not constitute formal assessment. Qualitative approaches are not the same as random responses and are designed to collect the same as random responses and are designed to collect data that can be verified and validated.data that can be verified and validated.

The Ford Foundation is interested in vigorous assessment that: The Ford Foundation is interested in vigorous assessment that: allows you to shape programs as they are implementedallows you to shape programs as they are implemented demonstrates what has been achieveddemonstrates what has been achieved enables you to examine what might be done differently as enables you to examine what might be done differently as you continue to develop programsyou continue to develop programs

ANALYSISANALYSISMIRRORS PROCESSMIRRORS PROCESS

INDUCTIVEINDUCTIVE ON GOINGON GOING MULTILAYEREDMULTILAYERED COMPLEXCOMPLEX ENRICHED THROUGH MEMBERS ENRICHED THROUGH MEMBERS

CHECK-INCHECK-IN

Qualitative ReportingQualitative Reporting

Reporting formats are shaped by assessment Reporting formats are shaped by assessment purpose and by the meaning that is being purpose and by the meaning that is being conveyed.conveyed.

Typical qualitative reporting includes narrative Typical qualitative reporting includes narrative accompanied by matrix or charts that accompanied by matrix or charts that demonstrate key issues or causal relationships.demonstrate key issues or causal relationships.

Ford Foundation assessment reporting seeks to Ford Foundation assessment reporting seeks to communicate outcomes, indicators of program communicate outcomes, indicators of program success and lessons learned through DDI success and lessons learned through DDI programs. Use of case studies and rich programs. Use of case studies and rich description is of particular interest.description is of particular interest.