Assessing a Campus-wide Internationalization Initiative: Aligning Assessments with

Institutional Outcomes

M. David Miller Timothy S. Brophy

University of Florida

Presented at Annual Meeting of the Association for the Assessment of Learning in Higher Education, Albuquerque, NM, June, 2014.

Quality Enhancement Plan - SACSCOC

Learning Without Borders: Internationalizing the Gator Nation• Based on institutional assessment• Focuses on learning outcomes and enhances

student learning environment• Includes broad-based involvement of

institutional constituencies in development and implementation

• Identifies initiatives to promote learning outcomes

• Assesses their achievement for the institution

Development of the Initiative – Broad Based Involvement

Internationalization Task Force (ITF) formed in January

2011

ITF has representatives

nominated by the Deans of units across campus

ITF involves all stakeholders –

faculty, students, administration,

staff and others

Alignment

Student Learning Outcomes

AssessmentCampus Initiatives

InternationalizationInternationalization is the conscious integration of intercultural and global

competencies into student learning.

GLOBAL CITIZENSHIP Students comprehend the trends, challenges, and opportunities that affect our communities and communities worldwide..

CONTENT:SLO 1. Students identify, describe, and explain global and intercultural conditions and interdependencies

CRITICAL THINKING:SLO 2. Students analyze and interpret global and intercultural issues

COMMUNICATION:SLO 3. Students communicate effectively with members of other cultures

INTERCULTURAL COMMUNICATIONS Students interact effectively with members of other cultures.

Initiatives

Curriculum Enhancement

Study Abroad

Campus Life

International Calendar and International Event Support

International Scholars Program

For students: new and exciting courses for the International Scholars Program

For faculty: support and professional development opportunities to develop International Scholar courses

Curriculum Enhancement

For students: more scholarships and exciting new programs, especially

in underrepresented disciplines

For faculty: support and professional development opportunities to develop Study Abroad programs

Study Abroad

Increase the number of events that bring together international and domestic students

Campus Life

Develop a plan for the university to

showcase the new international calendar and

specific events through various

media

Develop suggestions for

synergies to harness UF’s

current sources of international

events

International Calendar

Students earn

medallions at graduation to

recognize their

international expertise

International Scholars Program

Assessment Principles

Direct and indirect assessments must measure progress on the SLOs

Outputs must measure participation in the initiatives

Each measure must have a planned use or interpretation (Validity)

Each measure must be reliably measured (minimize error)

Using Existing Instruments

It is easier t

o a

dopt a

n existi

ng i

nstr

ume

nt t

ha

n t

o

devel

op a

ne

w i

nstr

ume

nt

Easier is not better if you violate your principles!

That is, existi

ng i

nstr

ume

nts

must:

meas

ure t

he SL

Os;

be vali

d f

or its

pla

nne

d

uses a

nd i

nter

pretati

ons; a

nd

be relia

ble f

or y

our

pur

poses

Experts for Judging Alignment (Content

Validation)• International Task Force

• Assessment Committee (5-6 with expertise on content and assessment)– Curriculum experts– International Administrators– Psychometricians

Defining Review Criteria for Existing Assessments

Measures the SLOs

Validity Evidence

Reliability Evidence

Feasibility to Use in Our

Context

Factors in our decision to create our own assessments

Most commercial products had good reliability and validity evidence for their stated purposes

None matched our SLOs (Alignment problem)

Little evidence of feasibility for large scale use

Final Decision: Need to develop our own assessments

What Assessments do we need to Develop?

Direct Assessments focusing on SLO achievement

Must allow flexibility to measure learning in any

discipline

Needs to be aligned with Curriculum

Indirect Assessments of SLOs 2 and 3 only

Not appropriate for SLO 1 since Content is discipline-

based

Developing the Assessments

Writing the Assessment Items to

measure SLOs

Ongoing review for fidelity with the SLOs

and Initiatives (Validity)

Piloting to establish reliability and psychometric

properties of items

Indirect Assessment Item Development

IntCRIT

1. I consider different perspectives before making conclusions about the world.

2. I am able to manage when faced with multiple cultural perspectives.

3. I am open to different cultural ways of thinking in any international context.

4. I can make effective decisions when placed in different cultural situations

5. Knowing about other cultural norms and beliefs is important to me.6. I am able to think critically to interpret global and intercultural

issues.7. I actively learn about different cultural norms.8. Understanding different points of view is a priority to me.9. I can recognize how different cultures solve problems.10. I can contrast important aspects of different cultures with my own.11. Knowing about other cultural beliefs is important.12. I am able to recognize how members of other cultures make

decisions

IntCOMM

1. I demonstrate flexibility when interacting with members of another culture. 2. I am confident that I can adapt to different cultural environments3. I am able to communicate effectively with members of other cultures4. I like working in groups with students from other countries.5. I feel comfortable in conversations that may involve cultural differences.6. When working on a group project, I enjoy collaborating with students from

other countries.7. I often ask questions about culture to members of other cultures.8. I enjoy learning about other cultures9. I appreciate members of others cultures teaching me about their culture.10. I am able to interact effectively with members of other cultures.11. I appreciate differences between cultures12. I feel comfortable discussing international issues.13. I can clearly articulate my point of view to members of other cultures

Direct Assessment

Rubrics developed for each of the three SLOs

Rubrics to be implemented in Curriculum and Study Abroad portion of the Initiative

Rubrics allow for discipline based content

Developing Rubrics

• The Association of American Colleges and Universities developed 15 Rubrics that can be used across programs and courses

• VALUE (Valid Assessment of Learning in Undergraduate Education) rubrics were developed by faculty and assessment expert teams across the country -- used by more than 2000 institutions (http://www.aacu.org/value/index.cfm)

QEP Content Rubric

SLOComponents

Outstanding3

Satisfactory2

Unsatisfactory1

Not Applicable0

Concepts/Principles

Consistently and effectively demonstrates sophisticated understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, and beliefs and practices.

Usually demonstrates understanding of the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.

Rarely or never understands the complexity of factors important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices.

Not Applicable to Assignment or Course

Terminology

Consistently recognizes and effectively utilizes important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.

Usually identifies and implements important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.

Rarely or never understands important and relevant terminology regarding intercultural and global issues in the appropriate environmental context.

Not Applicable to Assignment or Course

Methodologies

Consistently comprehends and effectively utilizes diverse and appropriate methodologies for understanding complex intercultural and global issues.

Usually comprehends and utilizes diverse and appropriate methodologies for understanding intercultural and global issues.

Rarely or never comprehends and utilize diverse and appropriate methodologies for understanding intercultural and global issues.

Not Applicable to Assignment or Course

Rubric Use

• Used for courses (curriculum) and study abroad

• Use must be approved by International Scholars Course Approval Committee

• Must adopt SLO(s) from program • Must show assignments and specific

tests that measure the rubrics• Otherwise, not part of International

Scholars program

Assessment PlanUniversity wide impact

Indirect Assessments

SERU Biennial, odd years

IntCrit and IntComm – Annual – Fall - Sampling

Outputs

Monitor participation and implementation - Annual

Impact on those taking courses or Study Abroad

Direct Assessments - Rubrics

Indirect Assessments – IntCrit and IntComm

Output – Number of students and options

For further information:

• See QEP online:http://sacs.aa.ufl.edu/sacs-reports• Contact us:

dmiller@coe.ufl.edutsbrophy@ufl.edu