54
Assessment Toolbox Assessment Toolbox for International for International Educators Educators Presenters: Presenters: David Comp, Darla K. David Comp, Darla K. Deardorff, Deardorff, Elaine Meyer-Lee, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki Lee Sternberg, Victor Savicki

Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Embed Size (px)

Citation preview

Page 1: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment Toolbox for Assessment Toolbox for International EducatorsInternational Educators

Presenters: Presenters: David Comp, Darla K. Deardorff, David Comp, Darla K. Deardorff,

Elaine Meyer-Lee, Elaine Meyer-Lee, Lee Sternberg, Victor SavickiLee Sternberg, Victor Savicki

Page 2: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Session OverviewSession Overview

Assessment Overview and ContextAssessment Overview and Context

Highlights of Specific Assessment Highlights of Specific Assessment Tools/MethodsTools/Methods

Tools/Methods Exploration through Tools/Methods Exploration through “Roundtable” discussions“Roundtable” discussions

Page 3: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessing International Assessing International Education: An OverviewEducation: An Overview

Dr. Darla K. DeardorffDr. Darla K. Deardorff

Duke UniversityDuke University

[email protected]@duke.edu

Page 4: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Why assess international Why assess international education?education?

Page 5: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Why assess international Why assess international education?education?

Quality improvementQuality improvement

AdvocacyAdvocacy

AccreditationAccreditation

Student learningStudent learning

Page 6: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Why Assess? Measure successWhy Assess? Measure success

Moving beyond Moving beyond

OUTPUTS to OUTCOMESOUTPUTS to OUTCOMES

What are meaningful outcomes What are meaningful outcomes (results) of internationalization (results) of internationalization

efforts?efforts?

Page 7: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

__________________________________________________________________________ 

INTERNATIONALIZATIONat institutions of higher education

 Inputs/Resources

needed for implementation of components of internationalization|

Activities/Components of Internationalization(college leadership, faculty involvement, curriculum, study abroad, international

students/scholars/faculty, international co-curricular units) |

Outputs of Internationalization(i.e., number of international students, number of study abroad programs, number of students

studying foreign languages, etc.)|

Outcomes of InternationalizationIntercultural competence – what is it?

How do higher education administrators define it? intercultural experts?How can it be assessed?

 = Long-Term Impact of Internationalization

_______________________________________________________________Figure 2. General program logic model applied to internationalization. (Deardorff, 2004, Page 58)

Page 8: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Ways of approaching Ways of approaching assessmentassessment

Some key points and Some key points and

lessons learned…lessons learned…

Page 9: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment-ContextsAssessment-Contexts

““At Home”At Home” – – curriculum, extra-curriculum, extra-curricular activities, community curricular activities, community involvement/impact, domestic/internat’l involvement/impact, domestic/internat’l student interaction, policy, financial, student interaction, policy, financial, campus climatecampus climate

AbroadAbroad (Cross-border)(Cross-border) – – involves involves cross-border delivery of education through cross-border delivery of education through exchange, distance and e-learning, branch exchange, distance and e-learning, branch campuses, partnerships, host community campuses, partnerships, host community impactimpact

Page 10: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment – Lessons LearnedAssessment – Lessons Learned

Collaborate - Put together Assessment Collaborate - Put together Assessment TeamTeam

Adapt - Build on what you haveAdapt - Build on what you have Measure – what is valued (align!)Measure – what is valued (align!) Plan - Develop assessment planPlan - Develop assessment plan Use – use what you assessUse – use what you assess Support – from leadership, Support – from leadership,

stakeholdersstakeholdersDeardorff, 2008Deardorff, 2008

Page 11: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

ASSESSING OUTCOMESASSESSING OUTCOMES

Starting point:Starting point:

Mission StatementMission Statement intointo

Goals Goals into into

Measurable ObjectivesMeasurable Objectives

Page 12: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment CycleAssessment Cycle

Define outcomes (based on Define outcomes (based on mission/goals) and establish measurable mission/goals) and establish measurable criteriacriteria

Identify appropriate assessment Identify appropriate assessment methodsmethods

Collect dataCollect data Analyze dataAnalyze data Reflect on needed changesReflect on needed changes Design and apply changesDesign and apply changes

Page 13: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment – Lessons LearnedAssessment – Lessons Learned

Multi-method, multi-perspectiveMulti-method, multi-perspective

OngoingOngoing

IntegratedIntegrated

Page 14: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Student Involvement Student Involvement in Assessmentin Assessment

Student outcome assessment is not a Student outcome assessment is not a measurement of learning but an measurement of learning but an integral integral part of learning. Can include part of learning. Can include language/disciplinary assessment, personal language/disciplinary assessment, personal growth, intercultural competence, etc. growth, intercultural competence, etc.

Involve students in negotiating Involve students in negotiating outcomesoutcomes

Page 15: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment MethodsAssessment Methods Methods determined by clearly Methods determined by clearly

articulated assessment questions – articulated assessment questions – what do we want to measure?what do we want to measure?

Include:Include: DIRECT METHODS – authentic assessment, DIRECT METHODS – authentic assessment,

portfolios, observationportfolios, observation

INDIRECT METHODS – surveys, focus groups, INDIRECT METHODS – surveys, focus groups, interviewsinterviews

COMBINATION OF QUANTITATIVE AND QUALITATIVECOMBINATION OF QUANTITATIVE AND QUALITATIVE

Page 16: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

DIRECT METHODSDIRECT METHODS

Embedded course assessmentEmbedded course assessment PortfoliosPortfolios PerformancePerformance TestingTesting Papers/projectsPapers/projects CapstoneCapstone

Page 17: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Indirect MethodsIndirect Methods

Surveys (inc. self-assessments)Surveys (inc. self-assessments) InterviewsInterviews Focus groupsFocus groups Curriculum/transcript analysisCurriculum/transcript analysis Documented dataDocumented data

Page 18: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

ICC Assessment Tools (85+!)ICC Assessment Tools (85+!)

Intercultural Development InventoryIntercultural Development Inventory Cross Cultural Adaptability InventoryCross Cultural Adaptability Inventory Intercultural Conflict Styles InventoryIntercultural Conflict Styles Inventory Language Strategies SurveyLanguage Strategies Survey Strategies Inventory for Culture LearningStrategies Inventory for Culture Learning Beliefs, Events, Values InventoryBeliefs, Events, Values Inventory GAP TestGAP Test Assessment of Intercultural CompetenceAssessment of Intercultural Competence

Page 19: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

ExamplesExamples

Georgia Tech’s Georgia Tech’s International Plan – International Plan – uses uses capstone, IDIcapstone, IDI

DukeDuke’s teacher ed program – ’s teacher ed program – uses self-report uses self-report instruments, embedded assessment, instruments, embedded assessment, faculty/supervisor observation, focus groupfaculty/supervisor observation, focus group

BCABCA uses embedded assessment, lang. tests, uses embedded assessment, lang. tests, BEVI, resident director and host family observationsBEVI, resident director and host family observations

ACEACE – – uses e-portfolio and self-report formuses e-portfolio and self-report form

Page 20: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Questions to consider…Questions to consider…

What are the specific goals and What are the specific goals and objectives?objectives?

What does the tool/method What does the tool/method specifically measure?specifically measure?

Limitations and cultural biases of Limitations and cultural biases of tools/methods?tools/methods?

Page 21: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

More questions to consider…More questions to consider…

How will assessment data be utilized? How How will assessment data be utilized? How will data be used to provide feedback to will data be used to provide feedback to students? To improve the students? To improve the program/curriculum, etc?program/curriculum, etc?

Is assessment multi-method, multi-Is assessment multi-method, multi-perspective, ongoing, intentional, perspective, ongoing, intentional, integrated, part of strategic plan?integrated, part of strategic plan?

Page 22: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

OVERVIEW OF OVERVIEW OF ASSESSMENT TOOLSASSESSMENT TOOLS

GPI, American Identity, GPI, American Identity,

SCAS, BEVISCAS, BEVI

Page 23: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment Toolbox for Assessment Toolbox for International Educators International Educators

– GPI– GPI

David CompDavid CompSenior Adviser for International Initiatives in The Senior Adviser for International Initiatives in The

CollegeCollegeThe University of ChicagoThe University of Chicago

[email protected]: Association of International EducatorsNAFSA: Association of International Educators

Annual ConferenceAnnual ConferenceWashington, D.C. ~ May 30. 2008Washington, D.C. ~ May 30. 2008

Page 24: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Global Perspectives Inventory (GPI)Global Perspectives Inventory (GPI)

Developed by Larry Braskamp, David Developed by Larry Braskamp, David Braskamp & Kelly Carter Merrill.Braskamp & Kelly Carter Merrill.

-Larry Braskamp is a senior fellow at the -Larry Braskamp is a senior fellow at the Association of American Colleges and Association of American Colleges and Universities (AAC&U). He is professor emeritus Universities (AAC&U). He is professor emeritus at the School of Education at Loyola University at the School of Education at Loyola University Chicago where he also served as senior vice Chicago where he also served as senior vice president for academic affairs for four years. president for academic affairs for four years. Braskamp also served as the first executive Braskamp also served as the first executive director of the Council for Higher Education director of the Council for Higher Education Accreditation (CHEA).Accreditation (CHEA).

Page 25: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Global Perspectives Inventory (GPI)Global Perspectives Inventory (GPI)

The GPI is a survey of 46 items specifically The GPI is a survey of 46 items specifically designed to provide self-reports of students’ designed to provide self-reports of students’ perspectives in three domains of holistic student perspectives in three domains of holistic student development--cognitive, intrapersonal, and development--cognitive, intrapersonal, and interpersonal. interpersonal.

The GPI can provide evidence of students’ global The GPI can provide evidence of students’ global perspectives:perspectives:

• At entry, during and at graduation from college At entry, during and at graduation from college • At entry and conclusion of an “education/study abroad” At entry and conclusion of an “education/study abroad”

experienceexperience

Braskamp, Braskamp & Carter Merrill, 2008Braskamp, Braskamp & Carter Merrill, 2008

Page 26: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Global Perspectives Inventory (GPI)Global Perspectives Inventory (GPI) Theoretical Background of GPITheoretical Background of GPI

• The construction of GPI is based on the developmental The construction of GPI is based on the developmental perspective that students are on a journey during perspective that students are on a journey during college. In this journey, students are given opportunities college. In this journey, students are given opportunities to reflect on three “big questions”:to reflect on three “big questions”:

““How do I know?” reflects the How do I know?” reflects the CognitiveCognitive dimension. dimension.

““Who am I?” reflects the Who am I?” reflects the Intrapersonal Intrapersonal dimension. dimension.

““How do I relate to others?” reflects the How do I relate to others?” reflects the InterpersonalInterpersonal dimension. dimension.

Braskamp, Braskamp & Carter Merrill, 2007Braskamp, Braskamp & Carter Merrill, 2007

Page 27: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Global Perspectives Inventory (GPI)Global Perspectives Inventory (GPI)Description of the six GPI Description of the six GPI

scalesscales

The The CognitiveCognitive domain consists of two domain consists of two scales: Knowing and Knowledge.scales: Knowing and Knowledge.

The The IntrapersonalIntrapersonal domain consists of two domain consists of two scales: Identity and Affect.scales: Identity and Affect.

The The InterpersonalInterpersonal domain consists of two domain consists of two scales: Social interactions and Social scales: Social interactions and Social responsibility.responsibility.

Page 28: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Global Perspectives Inventory (GPI)Global Perspectives Inventory (GPI)

Validity and ReliabilityValidity and Reliability

• ReliabilityReliability. Four GPI scales each loaded on their own independent . Four GPI scales each loaded on their own independent factors (some with the exception of one or two items) considering a factors (some with the exception of one or two items) considering a combination of both the highest and second highest loading scores: combination of both the highest and second highest loading scores: Cognitive/Knowing, Cognitive/Knowledge, Intrapersonal/Affect, and Cognitive/Knowing, Cognitive/Knowledge, Intrapersonal/Affect, and Interpersonal/Social Responsibility. The Intrapersonal/Identity and Interpersonal/Social Responsibility. The Intrapersonal/Identity and Interpersonal/Social Interaction scales did not seem to load onto any Interpersonal/Social Interaction scales did not seem to load onto any independent factor. The overlapping of scales on each of the factors is independent factor. The overlapping of scales on each of the factors is not troubling, but rather encouraging. Theoretically, scholars agree not troubling, but rather encouraging. Theoretically, scholars agree these factors would all contribute to one concept – global perspective. these factors would all contribute to one concept – global perspective.

• ValidityValidity. As students progress through college their development is . As students progress through college their development is likely to increase (Baxter Magolda, 1992; Pascarella & Terenzini, 2005). likely to increase (Baxter Magolda, 1992; Pascarella & Terenzini, 2005). Presumably, therefore, as people get older their global perspective Presumably, therefore, as people get older their global perspective may increase as well. may increase as well.

Braskamp, Braskamp & Carter Merrill, 2007Braskamp, Braskamp & Carter Merrill, 2007

Page 29: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Advantages and DisadvantagesAdvantages and DisadvantagesAdvantages

GPI can be administered at: entry, during and at graduation from college entry and conclusion of an “education/study abroad” experience

GPI has only 46 survey questions and brief demographic questions

GPI was developed by a leading expert in higher education assessment

Disadvantages

GPI is a relatively new assessment tool

GPI has not been used in any research studies

Page 30: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

New Tools for New Tools for Intercultural Learning Intercultural Learning

Outcomes AssessmentOutcomes AssessmentElaine Meyer-LeeElaine Meyer-Lee

DirectorDirector

Joy EvansJoy EvansAssistant Director for Research and ScholarshipAssistant Director for Research and Scholarship

Saint Mary’s College Saint Mary’s College Center for Women’s InterCultural LeadershipCenter for Women’s InterCultural Leadership

Notre Dame, Indiana Notre Dame, Indiana

Page 31: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

For five years we conducted a For five years we conducted a comprehensive study of our comprehensive study of our

intercultural and international intercultural and international learning on and off campus. learning on and off campus.

For study abroad we used a multi-For study abroad we used a multi-method longitudinal assessment of method longitudinal assessment of

intercultural engagement, IC intercultural engagement, IC sensitivity, identity, and goalssensitivity, identity, and goals

Page 32: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Evaluation & Assessment GoalsEvaluation & Assessment Goals

Evaluate programs’ effectivenessEvaluate programs’ effectiveness Measure Students’:Measure Students’:

• Changes in sensitivity to cultural differences Changes in sensitivity to cultural differences • American identity developmentAmerican identity development• Own sense of growth toward goalsOwn sense of growth toward goals

Explore correlations of change with:Explore correlations of change with:• Intercultural engagement while abroad Intercultural engagement while abroad

(interaction and reflection)(interaction and reflection)• Program or demographic characteristicsProgram or demographic characteristics

Page 33: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

American Identity MeasureAmerican Identity Measure

Adaptation of Jean Phinney’s Multigroup Ethnic Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record), and her Identity Measure (good track record), and her much newer American Identity Questionnaire much newer American Identity Questionnaire developed for underrepresented groupsdeveloped for underrepresented groups

Based on Eriksonian identity development theory. Based on Eriksonian identity development theory. Measures 2 Factors: Measures 2 Factors:

• identity search (e.g. I have often talked to other people identity search (e.g. I have often talked to other people about what it means to be an American.)about what it means to be an American.)

• affirmation, belonging, and commitment (e.g. Being affirmation, belonging, and commitment (e.g. Being American plays an important part in my lifeAmerican plays an important part in my life.).)

Page 34: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

American Identity MeasureAmerican Identity Measure Adapted it (with Phinney’s blessing) for study Adapted it (with Phinney’s blessing) for study

abroad students because one goal is more abroad students because one goal is more complexity in understanding one’s own culturecomplexity in understanding one’s own culture

Students in returnee courses felt this national Students in returnee courses felt this national identity version captured an important dimensionidentity version captured an important dimension

10-item Likert-style Self Assessment (5 for each)10-item Likert-style Self Assessment (5 for each)

Currently refining adaptation’s psychometric Currently refining adaptation’s psychometric properties, but alpha coefficients good so far properties, but alpha coefficients good so far (.79+)(.79+)

Page 35: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

SourcesSources

Phinney, J. (1992). The Multigroup Ethnic Phinney, J. (1992). The Multigroup Ethnic Identity Measure: A new scale for use with Identity Measure: A new scale for use with adolescents and young adults from diverse adolescents and young adults from diverse groups. groups. Journal of Adolescent ResearchJournal of Adolescent Research, , 77, , 156-176156-176..

Phinney, J., & Devich-Navarro, M. (1997). Phinney, J., & Devich-Navarro, M. (1997). Variations in bicultural identification Variations in bicultural identification among African American and Mexican among African American and Mexican American adolescents. American adolescents. Journal of Journal of Research on AdolescenceResearch on Adolescence, , 77, 3-32., 3-32.

Page 36: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Measure of Intercultural Measure of Intercultural EngagementEngagement

Asks students to quantify the Asks students to quantify the frequency of their engagement frequency of their engagement through:through:

Interaction with cultural resourcesInteraction with cultural resources Interaction with local peopleInteraction with local people Explicit reflection on these Explicit reflection on these

interactionsinteractions

Page 37: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

MEIMEI Comparable & Comparable &

program specific data program specific data on students’ on students’ intercultural intercultural interactionsinteractions

AIQAIQ Examine identity Examine identity

dimension, connect to dimension, connect to literature on thisliterature on this

? MEIMEI? Labor-intensive Labor-intensive

development and development and administrationadministration

? AIQAIQ ? Reliability and Validity of Reliability and Validity of

adaptation still being adaptation still being establishedestablished

? Needs refiningNeeds refining

Benefits to Benefits to

Study Abroad AssessmentStudy Abroad Assessment & Challenges to

& Challenges to

Institutional Research Institutional Research

Page 38: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Intercultural Leadership Intercultural Leadership Certificate ProgramCertificate Program

Capture Saint Mary’s strengths and CWIL’s unique location at the Capture Saint Mary’s strengths and CWIL’s unique location at the intersection of intercultural (local and global) and leadership intersection of intercultural (local and global) and leadership education in the context of a women’s collegeeducation in the context of a women’s college

Guide students through a developmental process into a deeper Guide students through a developmental process into a deeper capacity for intercultural leadership capacity for intercultural leadership

Bring coherence and provide an organizing framework for every Bring coherence and provide an organizing framework for every effort (curricular and co-curricular) being undertaken at Saint effort (curricular and co-curricular) being undertaken at Saint Mary’s College to encourage intercultural leadershipMary’s College to encourage intercultural leadership

Portfolio assessment of students’ proficiency as an intercultural Portfolio assessment of students’ proficiency as an intercultural leader in leader in 6 proficiency areas6 proficiency areas

Recognize the Leader WithinRecognize the Leader Within Articulate Your Ethical/Spiritual CenterArticulate Your Ethical/Spiritual Center Engage With & Value DiversityEngage With & Value Diversity Dialogue on Power & PrivilegeDialogue on Power & Privilege Create Inclusive & Equitable CommunityCreate Inclusive & Equitable Community Make Your Difference in the WorldMake Your Difference in the World

Page 39: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

For more handouts or bibliography on For more handouts or bibliography on assessing intercultural outcomes of assessing intercultural outcomes of study abroad programs, go tostudy abroad programs, go to http://www.saintmarys.edu/~cwil/php/intercultural.learning/IILOutcomes.php

or email me:or email me: [email protected] [email protected]

Page 40: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Assessment Toolbox for Assessment Toolbox for International Educators: International Educators:

SCASSCAS

Victor SavickiVictor Savicki

Western Oregon UniversityWestern Oregon University

Page 41: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Sociocultural Adaptation Scale Sociocultural Adaptation Scale (SCAS)(SCAS)

Developed by Colleen Ward and Developed by Colleen Ward and Antony KennedyAntony Kennedy

Ward, C. & Kennedy, A. (1999).Ward, C. & Kennedy, A. (1999). The Measurement of The Measurement of Sociocultural Adaptation.Sociocultural Adaptation. International Journal of International Journal of Intercultural Relations,Intercultural Relations, 2222, 659-677., 659-677.

Several subsequent supporting Several subsequent supporting studies..studies..

Page 42: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

ABC Theory of AcculturationABC Theory of Acculturation

Ward, C. (2001). The A, B, Cs of Ward, C. (2001). The A, B, Cs of acculturation. In D. Matsumoto (Ed.), acculturation. In D. Matsumoto (Ed.), Handbook of Culture and Psychology.Handbook of Culture and Psychology. (pp. (pp. 411-446). NY: Oxford University Press.411-446). NY: Oxford University Press.

AA Affect based on stress and coping theory Affect based on stress and coping theory BB Behavior based on learning cultural Behavior based on learning cultural

knowledge and skillsknowledge and skills CC Cognition based on social identification Cognition based on social identification

theory..theory..

Page 43: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Sociocultural Adaptation Sociocultural Adaptation DefinitionDefinition

SCAS measures the Behavior aspect SCAS measures the Behavior aspect of the ABC’sof the ABC’s

““The ability to ‘fit in,’ to acquire The ability to ‘fit in,’ to acquire culturally appropriate skills and to culturally appropriate skills and to negotiate interactive aspects of the negotiate interactive aspects of the host environment.”host environment.”

Applied to both foreign nationals Applied to both foreign nationals coming to the U.S and U.S. nationals coming to the U.S and U.S. nationals going abroad..going abroad..

Page 44: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

SCAS StructureSCAS Structure

29 items (reduced from an original pool of 41 29 items (reduced from an original pool of 41 items)items)• Making friends, Using the transport system, Going shopping, Making friends, Using the transport system, Going shopping,

Dealing with unsatisfactory service, Understanding cultural Dealing with unsatisfactory service, Understanding cultural differences, Understanding the local’s worldview, differences, Understanding the local’s worldview, Understanding the local value systemUnderstanding the local value system

Responses given on a scale rating the Responses given on a scale rating the difficulty the respondent has in dealing with difficulty the respondent has in dealing with each itemeach item. 1 (No difficulty) to 5 (Extreme difficulty. 1 (No difficulty) to 5 (Extreme difficulty))

Takes about 5 minutes to completeTakes about 5 minutes to complete Based-16 samples mostly from Australia, New Based-16 samples mostly from Australia, New

Zealand, and Singapore..Zealand, and Singapore..

Page 45: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

SCAS ScalesSCAS Scales

SCAS TotalSCAS Total: Sum/average of 29 items: Sum/average of 29 items Two sub-scales based on a factor analytic studyTwo sub-scales based on a factor analytic study

• Cultural Empathy and RelatednessCultural Empathy and Relatedness (13 items) (32.1% (13 items) (32.1% of variance)of variance)

Understanding local perspectives, values, world Understanding local perspectives, values, world views, and communicationviews, and communication

• Impersonal Endeavors and Perils Impersonal Endeavors and Perils (7 items) (8.7% of (7 items) (8.7% of variance)variance)

Management of impersonal interactions and/or Management of impersonal interactions and/or awkward situations..awkward situations..

Page 46: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Advantages and DisadvantagesAdvantages and Disadvantages

Based on a theory of acculturationBased on a theory of acculturation Tracks key intercultural adaptation issuesTracks key intercultural adaptation issues Gives several scores to interpretGives several scores to interpret Predicts an overall progression of Predicts an overall progression of

adaptationadaptation Can be used formatively and summativelyCan be used formatively and summatively List may be incompleteList may be incomplete List may have cultural bias..List may have cultural bias..

Page 47: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

The Beliefs, Events, and Values InventoryThe Beliefs, Events, and Values Inventory(BEVI)(BEVI)

Developed by Craig N. Shealy, Ph.D.Developed by Craig N. Shealy, Ph.D.James Madison UniversityJames Madison University

[email protected]

http://www.jmu.edu/ibavi

Page 48: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

The Beliefs, Events, and Values The Beliefs, Events, and Values Inventory Inventory

(BEVI)(BEVI)

What does the BEVI measure? What does the BEVI measure? The BEVI…The BEVI…is designed to assess a number of is designed to assess a number of

processes relevant to international experiences processes relevant to international experiences including (but not limited to): basic openness; including (but not limited to): basic openness; receptivity to different cultures, religions, and receptivity to different cultures, religions, and social practices; the tendency (or not) to social practices; the tendency (or not) to stereotype in particular ways; self and emotional stereotype in particular ways; self and emotional awareness; and preferred but implicit strategies awareness; and preferred but implicit strategies for making sense of why ‘other’ people and for making sense of why ‘other’ people and cultures ‘do what they do’ (from Shealy, 2005)cultures ‘do what they do’ (from Shealy, 2005)

Page 49: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

The BEVI:The BEVI:

Has been in development since the 1990sHas been in development since the 1990s Is derived from belief-value statementsIs derived from belief-value statements Is comprised of three validity and ten Is comprised of three validity and ten

process scalesprocess scales Does not appraise “correctness” of Does not appraise “correctness” of

individual items but an overall response individual items but an overall response pattern pattern

Is valid and reliable over timeIs valid and reliable over time

Page 50: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

The BEVI:

• Has 54 demographic items including age, gender, ethnic background, educational attainment, political orientation, and religious affiliation

• Has 391 items across a wide set of beliefs and values rated on a four-point Likert scale

• Takes 45-55 minutes to complete

• Is available online

Page 51: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

BEVI items include:

I am never sad.I have felt jealousy toward someone I loved.

There is no such thing as destiny. You can't argue with fate.

We should do more to help minority groups in our culture.Big corporations often harm the earth.

I know that evil people go to hell when they die.Hinduism and Buddhism have much to teach the modern world.

Page 52: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

The BEVI:

• Is effective in showing how people’s beliefs and values change as a result of having an international experience, including the way people feel about their own cultures, the value of learning about other

cultures, the importance of learning a second language, the value in learning about world events, and other attitudes.

• Measures complex phenomena and is not face-valid

• Administration time can be long for some

Page 53: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki
Page 54: Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

Your Turn…!Your Turn…!

Join us for “Roundtable Discussions” Join us for “Roundtable Discussions” on the specific tools you’ve just on the specific tools you’ve just heard about… ask more questions, heard about… ask more questions, learn more in detail about how and learn more in detail about how and when to use these instrumentswhen to use these instruments