Double-Loop Learning: A Multi-Institutional Effort to ...€¦ · with prior applications of the rubric to student work •Process of comparison allows for refining the rubric –(i.e

Double-Loop Learning: A Multi-Institutional Effort to Assess

Written Communication

Ruth Slotnick Mount Wachusett Community College

Susan Taylor Mount Wachusett Community College

Chris Cratsley Fitchburg State University

Nancy Schoenfeld Quinsigamond Community College

Gaelan Benway Quinsigamond Community College

Carol Lerch Worcester State University

NEEAN 16th Annual Fall Forum, Holy Cross, Worcester, MA, November 2, 2012

Outline of Presentation • Overview of Project

– Faculty Training

– Methodology

• Institutional Impact

– Worcester State University

– Mount Wachusett Community College

– Fitchburg State University

– Quinsigamond Community College

• Summary of Findings

• Next Steps

Model State-wide assessment systems

Institutional Assessment

Artifacts

Data

Data

Artifacts

Artifacts

Data


Common LEAP Assessment


of artifacts from all institutions

Aggregation of Data from all Institutions

Normalization and aggregation of Data from all Institutions

Assessment at the state level

Assessment at the institution level

Goals of Central Mass. Partnership to Assess Written Communication

• Compare assessment of Written Communication conducted using institutional rubrics from MWCC, Fitchburg State and QCC with assessment using the AAC&U VALUE rubric used by Worcester State.

• Determine how the assessment process and data can be used to improve teaching and learning, including establishing academic expectations for successful transfer between institutions.

• Evaluate the potential for value-added measurements of written communication and disaggregation of data as a function of individual student demographics.

Sample rubric alignment exercise Criteria Context of and

Purpose for

Writing

Content

Development

Genre and

Disciplinary

Conventions

Sources and

Evidence

Control of Syntax

and Mechanics

Quinsigamond 4 3 2 1 4 3 2 1 4 3 2 1 4 3 2 1 4 3 2 1

Format: Students

demonstrate awareness

of their audience and

purpose in written and

oral communication.

3

2

1

Organization: Students

organize their

presentation of

information with order

and structure.

3

2

1

Argument: Students use

appropriate modes of

both written and oral

communication.

3

2

1

Mechanics: Students

employ conventional

mechanics in their

presentation of

information.

3

2

1

Collecting Student Artifacts • Sample student writing artifacts from second

semester sophomore courses:

– Samples of 5-10 page papers from Sophomore level writing intensive courses collected Spring 2012

– Worcester State: Intro to Urban Studies – 22 artifacts

– MWCC: The Art of Being Human – 22 artifacts

– Fitchburg State: American Literature II – 27 artifacts

– QCC: Children’s Literature – 10 artifacts (others too)

• Sampling of student writing artifacts from first semester freshman writing courses is underway:

– Writing I, English Composition I, or its equivalent

Assessment Design

Worcester State artifacts (n=22)

MWCC artifacts (n=22)

Fitchburg State artifacts (n=27)

QCC artifacts (n=10 +)

VALUE rubric

Each assessed by 1 of 3 faculty from Worcester State and 1 faculty member from MWCC, Fitchburg or QCC

Each assessed by 1 faculty from Worcester State and one from MWCC.

Each assessed by 1 faculty from Worcester State and one from Fitchburg State.

Each assessed by 1 faculty from Worcester State and one from QCC.

MWCC rubric

8 assessed by 2 faculty from MWCC

Each assessed by 2 faculty from MWCC

FSU rubric 5 assessed by 2 faculty from Fitchburg State

Each assessed by 2 faculty from Fitchburg State

QCC rubric 9 assessed by 2 faculty from QCC

Each assessed by 2 faculty from QCC

Assessing Student Artifacts • All artifacts were uploaded into the Tk20 assessment

system at Fitchburg State and 3 faculty from each institution (12 total) were given usernames and passwords for the system.

• Each artifact was assessed electronically by 4 faculty members, 2 using an institutional rubric and 2 using the LEAP VALUE rubric.

• Faculty viewed the artifacts and entered scores electronically in Tk20, and had the assignment prompt which they received via e-mail.

• Data was aggregated in Tk20 and when necessary in Excel to examine differences between institutions and between rubrics.

Written Communication VALUE rubric Capstone - 4 Milestone - 3 Milestone - 2 Benchmark - 1

Context of and Purpose for Writing

thorough understanding, focuses all elements

adequate consideration, focus on the assigned task

awareness of, begins to show awareness of audience

minimal attention to, Instructor/self as audience

Content Development

illustrate mastery of the subject

explore ideas, shape the whole work

explore ideas through most of the work

develop simple ideas in some of the work

Sources and Evidence

skillful use, high quality, credible, relevant

consistent use, credible, relevant

attempt to use credible and/or relevant

attempt to use sources

Genre and Disciplinary Conventions

detailed attention to successful execution

consistent use of important conventions

Follows expectations appropriate to

Attempts to use consistent system

Control of Syntax and Mechanics

graceful language that skillfully communicates, virtually error free

straightforward language that generally conveys, few errors

language that generally conveys, some errors

language that sometimes impedes, errors

Worcester State Spring 2012 Data (n=22, each reviewed by 2 faculty with VALUE rubric)

0

5

10

15

20

25

30

35

40

45

Context andPurpose

Contentdevelopment

Sources andevidence

Genre andconventions

Syntax andmechanics

Pe

rce

nta

ge o

f ra

tin

gs

Written Communication Criteria

Capstone

Milestone 2

Milestone 1

Benchmark

Worcester State University • LEAP VALUE rubrics are being used to develop

programmatic and course outcomes

• Faculty are encouraged to use their own rubrics or LEAP VALUE rubrics in their courses

• Some faculty prefer other rubrics like the Writing Center Director’s Composition rubric

• In spite of many workshops focused on the use of rubrics there may still be some faculty resistance

• One question for discussion is whether the development or customization of rubrics at an institution helps facilitate faculty involvement

Comparing MWCC and VALUE rubrics MWCC rubric criteria LEAP VALUE rubric criteria

Establishes a Main Idea Context of and Purpose for Writing

Develops Support Content Development

Organizes Effectively (Partially covered by Genre and Disciplinary Conventions)

Establishes Purpose (Partially covered by Context and Purpose for Writing)

Uses Credible Research Material

Effectively and Ethically


Maintains Awareness of Audience Genre and Disciplinary Conventions (Partially covered by Context and Purpose for Writing)

Uses Appropriate Diction, Grammar and

Punctuation


MWCC Spring 2012 Data (n=22, each reviewed by 2 faculty with MWCC and VALUE rubrics)

0

10

20

30

40

50

60

70

80

90

100

Establishesmain idea

Developssupport

Organizeseffectively

Establishespurpose

Credibleresearch

Aware ofaudience

Pe

rce

nta

ge o

f ra

tin

gs


Yes

VALUE 2-4

No

VALUE 1

Appropriate mechanics

Mount Wachusett Community College • Rubric measures outcomes in terms of

“yes” or “no” – “No” column is pretty equivalent to LEAP benchmark

– “Yes” column does not break data out into ranges of competency, like the LEAP value rubric

– Data mostly indicates weaknesses, not how “good” some students are for each criterion

• Rubric language constraining – does not allow a variety of artifacts to be assessed

• For comparisons with other institutions, – more categories would also be useful

Comparing Fitchburg State and VALUE rubrics

Fitchburg State University rubric criteria LEAP VALUE rubric criteria

Controlling Idea


Development of Controlling Idea

Content Development

Organization (Partially covered by Genre and Disciplinary Conventions)



Academic Discourse Genre and Disciplinary Conventions (Partially covered by Context and Purpose for Writing)

Mechanics/Presentation


Fitchburg State Spring 2012 Data (n=27, each reviewed by 2 faculty with FSU and VALUE rubrics)

0

10

20

30

40

50

60

70

80

90

100

Controllingidea

Developmentof idea

Organization Sources andevidence

AcademicDiscourse

Mechanics/presentation

Pe

rce

nta

ge o

f ra

tin

gs


Proficient

VALUE 3-4

Sufficient

VALUE 2

Fitchburg State University • FSU institutional rubric strengths :

– Specificity of purpose for diagnosing student strengths and weaknesses

– Candor in the rating categories

• Data suggests: – At sophomore level, a sufficiently small enough number of

students are proficient to allow the rubric to assesses where students are with only 3 levels of ratings

• Additional categories: – Organization – Future plan to break sources and evidence into separate

categories for documentation and use of evidence allow for more diagnostic use of data

• Institutional modifications to the rubric also allow it to be used broadly across different assignment types

Comparing QCC and VALUE rubrics

QCC rubric criteria LEAP VALUE rubric criteria

FORMAT Students demonstrate awareness of their audience and purpose in written and oral communication.


ORGANIZATION Students organize their presentation of information with order and structure.

Content Development (and partially covered by Genre and Disciplinary Conventions)

ARGUMENT Students use appropriate modes of both written and oral communication (e.g., descriptive, informative, or persuasive).

Sources and Evidence (and partially covered by Content Development)

MECHANICS Students employ conventional mechanics in their presentation of information.


QCC Spring 2012 Data (n=10, each reviewed by 2 faculty with QCC and VALUE rubrics)

0

10

20

30

40

50

60

70

80

Format Organization Argument Mechanics Genre andconventions

Pe

rce

nta

ge o

f ra

tin

gs


Highest

VALUE 3-4

Intermediate

VALUE 2

Quinsigamond Community College • QCC sees the local development of the rubric useful

for community building and investment in the tool

• Student strengths and weaknesses were consistent with prior applications of the rubric to student work

• Process of comparison allows for refining the rubric – (i.e. adding criteria about plagiarism, and use of sources)

• QCC rubric more valuable as a teaching tool – Easier to communicate to students, with each cell

measuring a variety of things that are explicitly spelled out, so it is useful as a toolbox.

Comparing Spring 2012 Data (Each reviewed by 2 faculty with VALUE rubric)

0

0.5

1

1.5

2

2.5

3

Context andpurpose

Contentdevelopment

Sources andevidence

Genre andconventions

Syntax andmechanics

Me

an r

atin

gs


WSU

QCC

FSU

MWCC

Comparative Findings • Using both local and LEAP rubric:

– Students were mechanically strong and weaker in other areas, particularly in the use of evidence and integrating academic sources

• However:

– Application of LEAP VALUE rubric appears inconsistent across raters, suggesting that greater effort needs to be put into norming the rubric

• Major Finding:

– Substantial value to institutionally developed and modified rubrics, including students who see the investment of faculty working on these rubrics as pedagogical tools

Comparative Findings (continued) • Process of refining a rubric includes:

– Learning how to talk to each other about rubrics

– Struggling with when to change the rubric

– Embracing the philosophy that this is an ongoing process heavily reliant on a mixed methods approach, realizing that the qualitative aspect of norming is a time consuming yet rich data source

• Single loop learning is a technical change

• Double loop learning is an adaptive change

– The LEAP VALUE rubric is having adaptability problems, while the adaptive process of refining institutional rubrics is central to the professional development of faculty.

Next Steps

• Norming artifacts as a group to see if this process results in any meaningful statistical change

• Developing a system for collecting and comparing the qualitative data to code for emerging themes and patterns

• Measuring the goodness of fit of the artifact to the rubric both institutional and national

• Evaluate the potential for value-added measurements of written communication and disaggregation of data as a function of individual student demographics.

Model State-wide Assessment Systems


Artifacts

Data

Data

Artifacts

Artifacts

Data




of artifacts from all institutions

Aggregation of Data from all Institutions

Normalization and aggregation of Data from all Institutions

Assessment at the state level

Assessment at the institution level

Acknowledgments • Davis Educational Foundation

• Advancing Massachusetts Culture of Assessment (AMCOA) team – Peggy Maki, Neal Bruss, Bonnie Orcutt and Maureen Sowa

• Fitchburg State University – Anna Consalvo, Frank Mabee, and Joe Moser

• Mount Wachusett Community College – Julie Muszalski and Kara Roche

• Quinsigamond Community College – Valerie Clemente and Kathy Frederickson

• Worcester State University – Matt Mortoleva, Amber Vayo, and Amy West

Documents

Double-Loop Learning: A Multi-Institutional Effort to ...€¦ · with prior applications of the rubric to student work •Process of comparison allows for refining the rubric –(i.e