Indiana University

Scholarship of Teaching

and Learning

September 17, 2010

Establishing a Culture of Experimentation and Evidence

Reframing AssessmentTeaching as an Iterative

Process of Inquiry

Robert J. Thompson Jr.

Cultural Transformation is Occurring in the Academy• Acceptance of the need to do better

regarding undergraduate education

• Willingness to reexamine familiar practices and search for new methods

• Appreciation that experimentation to improve learning can be rewarding

Drivers of the Cultural Transformation

• Internal

• Faculty

• Institution

• Academy

• External – Society

• Internal External

Key Issues

• Focus on improving student learning

• Relationship between assessment and accountability

• Teaching as a process of inquiry

• Becoming a learning organization

ApproachesEngage societal & academy organizations

in addressing key issues

Establish collaborative projects

• Iterative T & L experiments

• Share approaches to teaching and learning

• Share assessment methods and measures

Form and engage networks in collaborative studies

Improving Student Learning

• Thousand wild flowers

• Increase the yield

• Accelerate the pace

• Engage more faculty

• Integrate disparate efforts/findings

• Scale

Engaging organizations

• AAC&U

• Reinvention Center

• Department of Education

AAC&U

• Learning Objectives

• Curriculum

• Pedagogy

• Assessment

Curriculum

• General education

• Coherent

Pedagogies of EngagementLearning Communities

Research/ Capstone/Thesis

Writing intensive courses

Experiential Learning

Internships

Study Abroad

Service Learning

The Reinvention Center

Consortium of research universities (60+)

Mission: to strengthen undergraduate education through networking and promoting the exchange of information and ideas and multi-campus collaborative projects.

www7.miami.edu/ftp/ricenter/

Assessment and Accountability

Department of Education:Spellings Commission Report: 2006

• Access

• Affordability

• Quality

• Accountability

Commission on Higher Education: Recommendations

• Provide more evidence of student achievement and institutional performance and make this evidence primary when judging academic quality.

• Make information easily understandable and readily accessible to the public.

• Develop various means to compare institutions regarding their success in student achievement and institutional performance.

• Establish threshold standards for collegiate learning.

Judith Eaton Change September/October 2007, pp. 18-19

Issue: Relationship between assessment and accountability

… if assessment becomes synonymous with

standardized testing, what will happen to assessment undertaken for the purpose of

guiding improvement in instruction,

curricula and student services?

Trudy Banta Peer Review Spring 2007, p. 12.

Premise:

“Assessment can and should be designed to deepen and strengthen student learning, not just to document it.”

Carol Geary Schneider Peer Review, Spring 2007, p.3

Marketplace Perspective on Accountability

• Education is seen as a product that is provided

• Focus is on student achievement and institutional performance outcomes

• Approach is comparative and competitive among institutions

Academic View of Accountability

• Education is seen as a process of enabling growth

• Focus is on instituting a culture of experimentation and evidence with regard to teaching and learning

• Approach is evaluative within institutions and collaborative among institutions in pursuit of best practices

Academic Model of Assessment

The primary functions of assessment are

to improve our efforts to promote the intellectual growth and personal

development of our students

Accountability Reframed

Institutions of higher education hold themselves accountable to their multiple constituencies for establishing continuous, systematic, iterative processes to improve the quality of teaching and learning.

Teaching as a Process of Inquiry

Process of Inquiry

• Make observations• Pose questions• Identify sources of information• Identify assumptions• Propose explanation/Formulate a hypothesis• Conduct an experiment or study• Analyze findings• Consider alternative explanations• Communicate findings

Boyer: 1990 Scholarship Reconsidered

• Scholarship of Discovery

• Scholarship of Integration

• Scholarship of Application

• Scholarship of Teaching

Becoming a Learning Organization

Derek Bok: Our Underachieving Colleges (2006)• Argued the need for continuous, systematic,

experimentation and evaluation

• Process of enlightened trail and error

• Encouraged funding agencies to support institutional efforts to establish systematic, continuous processes to improve undergraduate education rather than specific particular innovations

Systematic Improvement in Undergraduate Education in

Research Universities

The Teagle Foundation

The Spencer Foundation

Teagle and Spencer Project

• Goal: to foster a culture of experimentation and evidence for undergraduate education at research universities such that iterative approaches to curricular and pedagogical efforts to enhance student learning and engagement become the standard of practice for departments and programs responsible for undergraduate education.

Objective

• To seed a number of iterative curricular and pedagogical experiments aimed at improving undergraduate teaching and learning at research universities.

Objective

• To build a knowledge base about promising practices regarding effective teaching and learning processes in two core areas of intellectual skills central to a liberal education: writing and critical thinking.

Objective

• To promote a spread of effect of both best practices and the adoption of iterative approaches to undergraduate education within and across fields and institutions.

Collaborative Project

• Carnegie Mellon• Duke• Georgetown• Indiana• Penn State• UC-Berkeley• UC-Davis

• Illinois-Urbana- Champaign

• Kansas• Michigan• Nebraska-Lincoln• UNC-Chapel Hill• USC

University of Nebraska

• Developing a system for assessing student learning regarding 10 general education learning objectives

University of California-Davis

• Writing in large lecture courses

Carnegie Mellon University

• Using argument diagramming in freshmen writing course to promote critical thinking

Indiana University

The History Learning Project

“Decoding the Discipline”

Leah Shopkow, Arlene Díaz,

Joan Middendorf, and David Pace,

Kansas University

Evaluating the Cognitive Apprenticeship Model for

Improving Undergraduate Student’s Critical Thinking and Writing Skills

Andrea Greenhoot

Department of Psychology

University of Kansas

Collaborators

Dan BernsteinDirector, KU Center for Teaching Excellence

Jennifer Church-DuranAssociate Dean of Library Instruction

Terese ThonusDirector, KU Writing Center

Catherine WeaverAssistant Professor of Political Science

• The Goal• Improve critical thinking, application and

writing skills in undergraduate students• Model: Cognitive Apprenticeship

• Identification, modeling of expert-like processes

• Staged, scaffolded learning tasks• Students are supported by experts and peers

Kansas University Project

• What is the impact of the program on targeted skills in the course?• Systematic application of rubrics each semester• Course-specific pre- and post-tests of target skills

• What is the impact of the program on general skills not specific to course goals?• Application of a general intellectual skills rubric to

samples of student work• General dimensions of learning, but assessed in embedded

student work rather than stand alone testing• Do target skills generalize to new contexts?

• Pre- and post-test using Collegiate Learning Assessment (CLA)

Inquiries and Measurement

University of Michigan

The Impact of Meta-Cognitive Strategies

within Writing in the Disciplines

Naomi Silver, Matthew Kaplan, Deborah Meizlish, and Danielle LaVaque-Manty

University of Michigan: Aim

The goal:

To discover what types of interventions and pedagogical strategies will help students better understand not only course content, but also discipline-specific modes of thinking and writing.

Questions:

To what extent does engagement in

meta-cognitive activities bring students’ conceptions of thinking and writing in the disciplines closer to that of faculty?

To what extent does better understanding of these modes of thinking improve the students’ actual writing?

Assignments designed to:

• Expose student thinking

• Enable productive dialogue between instructors and students

Reflective process with writing assignment• Planning: How the assignment will develop

disciplinary thinking/writing

• Monitoring: Marginal comments on difficult/interesting moments in writing

• Evaluation: How did the assignment promote disciplinary thinking/writing and what they can apply to future assignments

Evaluation

• Rubrics for “think-like” psychologist and political scientist

• Pre-post student appraisals

• Students’ written responses to planning and monitoring prompts

• Pre-semester interviews and post-semester focus groups with instructors

Duke University

• Goal: Increase undergraduate research and honors theses

• Questions:

• How will faculty manage the increased work load?

• How will we know if our approach is successful?

The Thesis Assessment Protocol

Julie Reynolds

Department of Biology

Duke University

TAP Protocol

• Framework: Scientific peer review process

• Timeline for revisions

• Methods for soliciting and responding to feedback

• Assessment guidelines & rubrics for faculty

Methods for soliciting and providing feedback• Students write first draft

• Students solicit feedback

• Faculty provide feedback through rubric worksheet

• Students respond to feedback via worksheet

• Students write final paper

Feedback & Assessment Rubric

Quality of Research

1. Does the thesis represent the student’s significant research?

2. Is the literature review accurate and complete?

3. Are the methods appropriate, given the student’s research question?

4. Is the data analysis appropriate, accurate and unbiased?

Feedback & Assessment Rubric

Critical Thinking Skills

5. Does the thesis make a compelling argument for the significance of the student’s research within the context of the current literature?

6. Does the thesis skillfully interpret the results?

7. Is there a compelling discussion of the implications of findings?

Writing Skills8. Is the writing appropriate for the target audience?

9. Does the thesis clearly articulate the student’s research goals?

10. Is the thesis clearly organized?

11. Is the thesis free of writing errors?

12. Are the citations presented consistently & professionally throughout the text & in the list of works cited?

13. Are the tables and figures clear, effective, and informative?

Student response to feedback

Summary of readers comment/Reader

Student response

Location in revised thesis

Examples: 1. My Faculty Reader said she didn’t see the relevance of the article by Smith and Jones (2002) to my research.

I rewrote the introduction to the paragraph in which I reviewed Smith and Jones’ research, making it more explicit that this research influenced the choice of methods that are commonly used in this field.

Literature review (in Introduction)

2. My Research Supervisor said he didn’t think I needed to provide so many background details in the Introduction.

I discussed this with my Faculty Reader who said that as an outside reader, she appreciated the extended background section. So, I decided to keep all the details I presented in the background section, but to revise it for conciseness.

Introduction

Teagle/Spencer Project Findings

• Approaches – insights regarding

what and how

• Findings – student learning

Critical Thinking

• Disciplinary specific

• Explicate the metacognitions

• Provide the scaffolding

Curriculum: From General Education to Education in the Disciplines:

“if we improve learning in the disciplines, we will have improved general education”

Beyer, Gillmore, & Fisher, 2007, p. 363)

Writing

• Makes thinking visible

• Enables dialogue about how to think more critically

• Engages metacognitive processes:

• Planning,

• Monitoring

• Evaluation

Learning to Write & Writing To Learn

• Assignments

• Feedback > revision

• regarding what – targets

• how – methods Rubrics

• Grading

• criteria

• how - methods

Teagle/Spencer Project Conference

• What does evidence of metacognition look like on your campus?

• How does it differ across disciplines?

• How does it differ across levels of student development?

• What are the challenges to implementation of strategies related to metacognition?

Teagle/Spencer Project Conference

• What is the relationship between assessment and faculty development?

• Is either one a good and useful wedge into the other?

• Is assessment a teachable moment for good practice?

• Is department level assessment a route to participation in teaching improvement?

Establishing a Culture of Experimentation and Evidence

Next Steps

“ Just as weighing a pig will not make it fatter, …” (p.10).

Banta, T. W. (2007). Can assessment for accountability complement assessment for improvement? Peer Review 9 (2), p. 9-12.

National Center for Post Secondary Improvement• Extensive involvement in student

assessment but limited use and impact

• “..student assessment data has only a marginal influence on academic decision making.”

• Research universities make the least use of student assessment data for academic decisions

Correlates of Use

• The extent of student assessment being conducted

• The extent of professional development related to student assessment for faculty, staff, and administrators

(Peterson & Augustine, 2000, Research in Higher Education, 41, pp.21-52; 443-479)

National Institute for Learning

Outcomes Assessment• Established in 2008, (NILOA) assists

institutions and others in discovering and adopting promising practices in the assessment of college student learning outcomes.

www.learningoutcomeassessment.org

NILOA’s primary objective:

• to discover and disseminate ways that academic programs and institutions can productively use assessment data internally to inform and strengthen undergraduate education, and externally to communicate with policy makers, families and other stakeholders.

NILOA Report

More Than You Think, Less Than We Need:

Learning Outcomes Assessment in

American Higher education

(Kuh & Ikenberry, 2009)

Key Points• Major drivers for assessment:

• Accreditation

• Commitment to improvement

• Must Provide evidence of how findings are being used to improve student learning

• Program level findings are more likely to be actionable

• Know what to do to improve student learning

Occasional Papers

• Ewell, P. (2009, November). Assessment, accountability, and improvement: Revisiting the tension.

• Banta, T.W., Griffin, M., Flateby, T.L., & Kahn, S. (2009, December).Three promising alternatives for assessing college students' knowledge and skills.  

Occasional Papers

• Wellman, J.V. (2010, January). Connecting the dots between learning and resources.

• Hutchings, P. (2010, April). Opening doors to faculty involvement in assessment.

• Swing, R.L. & Coogan, C.S. (2010, May). Valuing assessment: Cost-benefit considerations.

Engage Networks

Reinvention Center

Assessment Working Groups• General education• E-portfolios and rubrics• Writing to learn in the STEM disciplines

• NSF supported Workshop• Multicampus collaborative proposal