Development of an Assessment Strategy for Global Learning Outcomes at Koç University

Development of an Assessment Strategy for Global Learning Outcomes at Koç

University

PART

I. Become aware of the importance of assessing global learning outcomes

II. Approach to meaning and goals of global learning and competencies

III. Discover methodological possibilities with Samples

IV. Get recommendations

AGENDA

Part I.Become aware of the importance

of assessing global learning outcomes

“It was a life changing experience!”

Assessment is no goal but a means to an end.

Assessment as instrument• See effectiveness of internationalization strategy

General Program Logic Model Applied to InternationalizationSource: Darla Deardorff (2004)

Assessment as instrument• See effectiveness of internationalization strategy

-> Office Assessment

• Promote programs -> Marketing

• Guide the learner

• Evaluate the significance of program type and length

• Enhance programs

Our overarching goal:

High Quality Education

Winthrop University – a pionier of assessing global learning outcomes

• Multiple methods • Key component: Questionnaire that measures changes in

global knowledge, attitudes, and engagement- Self developped - Can be administered frequently without additional cost- Students’ responses can be tracked over time through their university ID numbers

• Additional Qualitative Methods:- Student’s evaluation of global peer mentor- Global Learning cultural event reflective papers- Online presentation- Students reflective papers- Assessment of University-Level Competencies etc.

Part II.Approach to meaning and goals

of global learning and competencies

Part II.1.Determine Goals

Define Goals

By knowing where you intend to go, you increase the chances of you and

the learner ending up there.

Define Goals –

5 levels to keep in mind

Establishing Global Learning GoalsSource: Association of American Colleges and Universities (2006): Assessing Global Learning – Matching Good Intentions with Good Practice

Sample 1: Definition of Global Learning Outcomes at Webster University

Sample 2: Definition of Global Learning Outcomes at Winthrop University

Sample 3: Proposal of the Association of American Colleges and Universities

Part II.2.Define

Competences

“[...]competence can be measured. But its measurement depends first on its definition.”

Klemp, 1979

Define Competences

Use definitions that already exist Subdivide Integrate levels Define Indicators

Sample 1: Proposal of the Association of American Colleges and Universities (2009) – Definition of expected competences

Sample 1: Proposal of the Association of American Colleges and Universities (2009) – Definition of expected competences

Sense of global interconnections and interdependencies

Consider individuals intervention in a global social problem possible

Ability to describe a social problemAbility to identify some of the ethical

and moral questions that underlie a given transaction between countries

Development of greater courage to engage in social exchanges and enterprises, even when faced with radical cultural difference

Identification of obligations to people situated both inside and outside their own national borders

Sample 2: Definition of Global Learning Outcomes at Webster University

Intercultural Competence

Cultural Self-Awareness (Knowledge)Cultural Worldview Frameworks (Knowlegde)Empathy (Skills)Verbal and Nonverbal Communication (Skills)Curiosity (Attitudes)Openness (Attitudes)

Sample 2: Definition of Global Learning Outcomes at Webster University

Cultural Self-Awareness (Knowledge)Cultural Worldview Frameworks (Knowlegde)Empathy (Skills)Verbal and Nonverbal Communication (Skills)Curiosity (Attitudes)Openness (Attitudes)

BeginningDevelopingProficientExemplary

Intercultural Competence

Sample 2: Definition of Global Learning Outcomes at Webster University

Intercultural Development Continuumby M. Hammer, IDI & Milton Bennett (1993)

Part III.Methodological

Possibilities

• Developed by intercultural experts with a developmental focus

• Paper based quantitative assessment tool: pre-posttest

• Independently reviewed and tested• Strong reliability & validity• Available in several languages• Designed in 2010 and already used by over 150

Institutions and Universities

Global Perspective Inventory

Global Perspective Inventory

Abilene Christian University, Agnes Scott College, Alderson-Broaddus College, American Institute for Foreign Study(AIFS), Anderson University, Appalachian State University, Arkansas State University, Auburn University, Augstana College, Illinois, Baldwin Wallace College, Banard College, Bellarmine University, Bethany College (WV), Bethel University (MN), Boise State University, Brevard College, Brigham Young University, Brown University, Buena Vista University (IA), Campbellsville College, Carroll University (WI), Carson Newman College, CEA Global Education, Center for Cross-Cultural Study (Spanish Studies), Central College (IA), Central College Abroad, Central Lakes College (MN), College of the Mainland, College of Wooster, Colombia College (IL), Colorado State University, Concordia College (MN), Davis and Elkins College, Delta University, Des Moines University, Drake University, Duke University, Elgin Community College, Elmhurst College, Elon University, Emory University, Emory and Henry College, Fairfield University, Ferrum University, Florida International University, Florida State University, Fordham University, Global Citizen Year, Grove City College, Guilford College, Gustavus Adolphus College, Hampton University, Hartwick College, Hobart and William Smith College, Hope College, Illinois State University, Indiana University, Indiana Wesleyan University, Johnson University, Kalamazoo College, King College, LaFayette College, Lee University, Lees McRae College, Lenoir Rhyne College, Lesley University, Lincoln Memorial University, Lindenwood College, Lindsey Wilson College, Loyola University Chicago, Lynn University, Macalester College, Marquette University, Marshall University, McDaniel College, Menominee Nation College, Mercer University, Miami Regional University (OH), Michigan State University, Mississippi State University, Milligan College, Monterrey Technological University (Mexico), New York University, North Carolina State University, Northeastern University, Ohio Valley University, Oregon University System, Pacific Lutheran University, Peninsula College, Pitzer College, Portland State University, Queens University, Randolph Macon College, Regent University, Santa Barbara City College, Santa Fe College, Semester at Sea, Seminole State College, Shenandoah University, Smith College, South Central University, St Ambrose College, St Augustine College, St Bonaventure University, St Cloud State College, St Edwards University, St Lawrence University, St Louis Community College, St Mary's College, SUNY Oswego, Susquehanna University, The Chicago School of Professional Psychology, Texas A & M University, Texas Woman's University, Tulane University, Tusculum College, University of Charleston (WV), University of Cumberlands, University of Georgia, University of Illinois Chicago, University of Maryland, University of Michigan, University of North Carolina, Chapel Hill, University of Northern Iowa, University of Notre Dame, University of Pikeville, University of Pittsburgh, University of San Diego, University of South Carolina Upstate, University of Southern Indiana, University of St Thomas (MN), University of Tennessee, Knoxville, University of Texas, San Antonio, University of Texas, Tyler, University of Wisconsin, Eau Claire, University of Wisconsin, La Crosse, University of Wisconsin, Milwaukee, University of Wisconsin, Platteville, Virginia Intermont College, Viterbo University, Walsh University, Washington University, Westminster College (UT), West Virginia Wesleyan College, Wofford College, Yale University

Abilene Christian University, Agnes Scott College, Alderson-Broaddus College, American Institute for Foreign Study(AIFS), Anderson University, Appalachian State University, Arkansas State University, Auburn University, Augstana College, Illinois, Baldwin Wallace College, Banard College, Bellarmine University, Bethany College (WV), Bethel University (MN), Boise State University, Brevard College, Brigham Young University, Brown University, Buena Vista University (IA), Campbellsville College, Carroll University (WI), Carson Newman College, CEA Global Education, Center for Cross-Cultural Study (Spanish Studies), Central College (IA), Central College Abroad, Central Lakes College (MN), College of the Mainland, College of Wooster, Colombia College (IL), Colorado State University, Concordia College (MN), Davis and Elkins College, Delta University, Des Moines University, Drake University, Duke University, Elgin Community College, Elmhurst College, Elon University, Emory University, Emory and Henry College, Fairfield University, Ferrum University, Florida International University, Florida State University, Fordham University, Global Citizen Year, Grove City College, Guilford College, Gustavus Adolphus College, Hampton University, Hartwick College, Hobart and William Smith College, Hope College, Illinois State University, Indiana University, Indiana Wesleyan University, Johnson University, Kalamazoo College, King College, LaFayette College, Lee University, Lees McRae College, Lenoir Rhyne College, Lesley University, Lincoln Memorial University, Lindenwood College, Lindsey Wilson College, Loyola University Chicago, Lynn University, Macalester College, Marquette University, Marshall University, McDaniel College, Menominee Nation College, Mercer University, Miami Regional University (OH), Michigan State University, Mississippi State University, Milligan College, Monterrey Technological University (Mexico), New York University, North Carolina State University, Northeastern University, Ohio Valley University, Oregon University System, Pacific Lutheran University, Peninsula College, Pitzer College, Portland State University, Queens University, Randolph Macon College, Regent University, Santa Barbara City College, Santa Fe College, Semester at Sea, Seminole State College, Shenandoah University, Smith College, South Central University, St Ambrose College, St Augustine College, St Bonaventure University, St Cloud State College, St Edwards University, St Lawrence University, St Louis Community College, St Mary's College, SUNY Oswego, Susquehanna University, The Chicago School of Professional Psychology, Texas A & M University, Texas Woman's University, Tulane University, Tusculum College, University of Charleston (WV), University of Cumberlands, University of Georgia, University of Illinois Chicago, University of Maryland, University of Michigan, University of North Carolina, Chapel Hill, University of Northern Iowa, University of Notre Dame, University of Pikeville, University of Pittsburgh, University of San Diego, University of South Carolina Upstate, University of Southern Indiana, University of St Thomas (MN), University of Tennessee, Knoxville, University of Texas, San Antonio, University of Texas, Tyler, University of Wisconsin, Eau Claire, University of Wisconsin, La Crosse, University of Wisconsin, Milwaukee, University of Wisconsin, Platteville, Virginia Intermont College, Viterbo University, Walsh University, Washington University, Westminster College (UT), West Virginia Wesleyan College, Wofford College, Yale University.

Florida State University – GPI Pretest-Posttest

Florida State University – GPI Pretest-Posttest

Some people have a culture and others do not.

The role of student is to receive knowledge from authority figures.

Much more Quantitative Instruments to Assess Learning Outcomes

• Global Competency Inventory (GCI), Kozai Group • Intercultural Effectiveness Scale (IES), Kozai Group • The International Profiler (TIP), WorldWork • Cultural Perspectives Questionnaire (CPQ),

International Institute for Management Development

• Self-Assessment for Global Endeavors (SAGE), Caligiuri and Associates

• ...

Photo ContestCategory Description

Knowledge In what ways do you better understand the country’s people, history, religion, politics, or culture?

FlexibilityIn what ways did you integrate into local life?In what ways did you adapt yourself to be comfortable with the locals?

Curiosity In what ways were you shocked, intrigued, or inspired to learn more about the people and the culture?

SkillsIn what ways did you gain new skills, such as resourcefulness, problem-solving, leadership, communcation, and language skills?

Growth

In what ways are you better prepared to be a global citizenship because of studying abroad?In what ways are you different as a result of your experience?

: Categories at TCU

Winner in the category “knowledge”

Quantitative Methods

• Pretest – Posttests• Secondary Data Analysis• Surveys by other campus units

BUT: Only in combination with

qualitative Methods!

Qualitative Methods – Self-Assessment

• Student presentations• Student portfolios• Professor evaluations • Narrative diaries / journals• Essays• Reflection papers

Qualitative Methods – Other Reported Instruments

• In depth student interviews• Faculty observation of students• Case studies• Audio that demonstrates

foreign language competency

Qualitative Methods - Reciprocative

• Focus groups• Dialogues• Workshops• Reflection Courses

Pre-Measurements

Encouraged student awareness of the learning outcomes prior to

participation and during the program.

Even more

... thematic learning communities (or linked courses); service- and community-based learning; experiential learning; vertically integrated curricula with first-year seminars and senior capstones; teaching science through problem-based inquiry; undergraduate research and active, hands-on, collaborative learning; and inquiry-based pedagogies...

Part IV.Recommendations

Step by Step

Koç University’s Mission, Core Purpose, Core Values,

Strategic Goals, and Initiatives

Departamental Mission Statement

Goals

Objectives / Outcomes

MeasuresAnalysis / Results

Recommendations / Improvement

Resource Impact

Following: A Model for Continuous Administrative Unit Improvement of Southwestern University

Developing Methodology -Keep in Mind:

combine qualitative and quantitative

methods

financial, technical and human resource

conditions

involve whole campus community incl.

students

Assessment should be ongoing and

multiple

Students’ individualityPrograms’ duration and type

Outcomes’ components and levelstemporally

Global Learning is a multi dimensional construct!While assessing...

Differentiate!

• Astin; Banta; Cross; El-Khawas; Ewell; Hutching; et al. (1992): Nine principles of good practice for assessing student learning. American Association of Higher Education Assessment Forum. Retrieved 06/04/13 from http://www.fctel.uncc.edu/pedagogy/assessment/9Principles.html

• Braskamp; Braskamp; Merril (2005): Global Perspecives Inventory. In: International Educator May/Jun 2005, p. 26-31.• Braskamp; Braskamp; Merrill (2009): Assessing Progress in Global Learning and Development of Students with Education Abroad Experiences. In: Frontiers: The

Interdisciplinary Journal of Study Abroad, v. 18, p. 101-118. • Bennett, M. (1993). Towards ethnorelativism: A developmental model of intercultural sensitivity. In R. M. Paige (Ed.): Education for the intercultural experience, p. 21–71. • Deardorff, Darla K. (2005): A Matter of Logic? In: International Educator, v. 14, p. 26-31. • Deardorff, Darla K. (2006): The Identification and Assessment of Intercultural Competence. In: Journal of Studies in International Education, v. 10, p. 241-266. • Doyle, Dennis (2009): Holistic Assessment and the Study Abroad Experience. In: Frontiers: The Interdisciplinary Journal of Study Abroad, v. 18, p. 143-155.• Dwyer, Mary M. (2004): More Is Better: The Impact of Study Abroad Program Duration. In: Frontiers: The Interdisciplinary Journal of Study Abroad, v. 10, p. 151-163.• Florida State University: Global Learning for Global Citizenship. Florida 2010. Retrieved 06/04/13 from

http://goglobal.fiu.edu/SiteCollectionDocuments/QEP_Report-Final.pdf• Franklin, K. U. (2010). Long-term Career Impact & Professional Applicability. Frontiers: The Interdisciplinary Journal of Study Abroad, v. 19, p. 169-190.• Ingraham; Peterson (2004): Assessing the Impact of Study Abroad on Student Learning at Michigan State University. In: Frontiers: The Interdisciplinary Journal of Study

Abroad, v. 10, p. 83-100.• Kehl; Morris (2007): Differences in global-mindedness between short-term and semester-long study abroad participants at selected private universities. Frontiers: The

Interdisciplinary Journal of Study Abroad, v. 15, p. 67-80.• Klemp, George (1979): Identifying, Measuring and Integrating Competence. In: Pottinger / Goldsmith (Eds.): Defining and Measuring Competence. San Francisco.• Knight, Jane (1997): A shared vision? Stakeholders Perspectives on the Internationalzation of Higher Education in Canada. Journal of Studies of Studies in International

Education in Canada, 1 (1), p. 27-44.• Mandernach, B. J. (2003): Writing Quality Learning Objectives. Retrieved 06/25/13 from http://www.park.edu/Old/cetl2/quicktips/writinglearningobj.html.• McTighe Musil, Caryn (2009): Assessing Global Learning – Matching Good Intentions with Good Practice. Washington.• Sobania; Braskamp: Study Abroad or Study Away: It’s not merely semantics. In: Peer Review, v. 11, no. 4, p. 23-27. • Southwestern University (2011): Administrative Assessment Handbook. Retrieved 06/13/13 from

http://www.southwestern.edu/offices/planning/assessment/support.php• Texas Christian University: Photo Contest. Retrieved 06/21/13 from http://www.studyabroad.tcu.edu/default.asp?pid=572• Van de Berg; Balkcum; Scheid; Whalen (2004): The Georgetown University Consortium Project: A Report at the Halfway Mark. Frontiers: The Interdisciplinary Journal of

Study Abroad, v. 10, p. 101-116.• Williams, Tracy (2009): The reflective model of intercultural competency: a multidimensional, qualitative approach to study abroad assessment. Frontiers: The

Interdisciplinary Journal of Study Abroad, v. 18, p. 289-306.• Webster University: Global Citizenship Program – Rubrics. Retrieved 06/04/13 from http://www2.webster.edu/globalcitizenship/documents/index.shtml• Winthrop University (2011): Quality Enhancement Plan – The Global Learning Initiative. Retrieved 06/12/13 from http://www2.winthrop.edu/gli/wuPublicGli.pdf

References