Comparative Revision Writing: Quantitative & Qualitative Research on Students’ Perspectives of Multimodal Instructor Feedback

Way JengTialitha Macklin

Mark TrianaStacy Wittstock

10:50-11:05 – Introductions & Ti Macklin11:05-11:10 – Questions11:10-11:30 – Mark Triana & Stacy Wittstock11:30 -11:35 – Questions11:35-11:50 – Way Jeng11:50-11:55 – Questions11:55-12:05 – Overall Questions

Panel Overview

Study Context

Response du Jour

Methodso Fall 13 - Spring 14o First-Year Composition

o 143 Student Participants

o 3 Teacher Participantso 9 Sections

o Written Control Groupso Same Assignmentso Same Response Style

o Free Software for Teachers & Studentso Adobe Readero Dropboxo iAnnotate

o Pre- & Post-Study Surveyo Collected One Set of

Essays

Student Response Preferences

Margins End Combo Con-fer-

ences

Audio0

0.1 75

0.3 5

0.5 25

0.7

11%3%

55%

32%

0%

17%

2%

40%

26%

14%10%

3%

64%

23%

0%

Pre-Study (N=120) Post-Study Audio (N=42)

Teacher Influence on Preferences

10%

30%

10%

50%

9%

63%

23%5%

40%

10%

50% Audio Group

21%

79%

21%

65%

14% 4%4%

60%

32%

Written Group

Teacher 2Teacher 1 Teacher 3

Student Narrative Feedback

Reasons for Rejecting Audio Responseo Unfamiliaro Learning Styleso Prevalence of Global

Commentaryo Timeo Technical Difficulties

Reasons for Selecting Audio Responseo Clarityo Learning Styleso Prevalence of Global Commentary

Overall Finding

But What About Revision?

Methodso Fall 15 - Spring 16o Same Data Set as Previous

Study (minus a few)o 3 Comp Teachers (not part

of data set)o 90 Essay Sets (redacted,

txt files)o Removed Citation Pageso Inter- Rater Reliability (4

essay sets)

o Trainingo Google Docso Text Compareo CWPA Granto Faigley, L., & Witte, S.

(1981). Analyzing Revision. College Composition and Communication, 32(4), 400–414.

Meaning Classifications• Surface Change: No change to meaning at all. All Formal change types are

Surface changes.• "You pay two dollars" → "You pay a dollar entrance fee.”• "Because the horse lost his shoe the rider was lost." → "Because the

horse lost his shoe, the rider was lost."

• Local Change: These changes are fairly small. While they add, remove, or alter information, the basic idea of the text remains the same. A summary description of the text would not change, based on this revision.

• Global Change: These changes are substantial. They may have an effect on the text globally. A summary of the text would change, based on this revision.

Type Classifications- Addition: Make explicit what could earlier be inferred or was unknown- Removal: Make implicit what was earlier explicit or make missing entirely - Substitutions: Re-phrasing by using different words- Permutations: Re-phrasing by using the same words- Distributions: Material in one text unit is passed into more than one- Consolidations: Material in multiple text units is compressed into fewer units- Move: Material is moved whole- Formal: Changes that affect only the form of text, and do not have the

potential to affect fundamental meaning. Often, these will be textual features associated with a style guide or grammar rule

- Grammar: Grammatical elements (e.g., clause structure, punctuation) is changed – syntactic structure

Coding Guide

Surface Local GlobalAddition SA LA GARemoval SR LR GRSubstitution SS LS GSPermutation SP LP GPDistribution SD LD GDConsolidation SC LC GCMove SM LM GMGrammar SG LG GGFormal SF

Rater Experience&

Pedagogical Implications

Coding Methodology- Skimmed through each essay in the set- Understanding the general gist of each essay helped with detecting

Global changes

- Worked in small, manageable samples, going paragraph by paragraph in Text-Compare

- Worked consistently on the project as much as possible- Long gaps between working on the project made it more likely that I

might change how I coded something

- Explained difficult coding decisions - I felt that this might help the researchers understand coding

differences between raters

Text-Compare

Inter-Rater Reliability

- Similar institutional backgrounds- All raters taught in the same program at WSU- All raters were either current or former graduate students at WSU- All raters experienced the same professional and teacher

development efforts at WSU

- Training also fostered uniformity- Practiced coding sample essay sets together- Encouraged discussion of disagreements over coding decisions

Impact on Assessment

- Feedback during the semester—increased focus on Global revision through screencast feedback- Less is more!

- Teaching revision—increased focus on helping students to understand the difference between Global vs. Local vs. Surface changes- Quality vs. Quantity in revision work

- Assessment—looking for Global vs. Local vs. Surface changes- Did the student actually revise? Or did he/she proofread?- What did I give feedback on? Basing assessment on what I actually

commented on.

Cultivating Peer Review Literacies- Establishing a complex yet manageable vocabulary- What is the value (or potential) of asking students to itemize,

compartmentalize, and classify their individual revisions as well as those made by their peers?

Cultivating Peer Review Literacies- Putting distinctions between individual revisions and

personalized revision strategies into practice- How might we go about designing more focused, formative, and

diverse writing and peer review activities that target particular kinds of revision (global, local, and surface)?

Cultivating Peer Review Literacies- Constructing peer review ecologies predicated on reflection and

meta-cognition- To what extent can we frame peer review as a complex genre of

writing, governed as much by rigid conceptual frameworks as subjective or holistic responses to writing as a whole?

Statistical Analysis&

Overall Findings

A Brief Note About Inter-Rater Reliability

This study used 3 different coders who looked at essays to produce data- What if they disagreed?- Inter-rater reliability (agreement) was measured by having all 3

raters look at several essays, then comparing their coding to each other- Numerically, this is reported as Cronbach’s Alpha, .962- Generally consider a very high value

- Intraclass correlation measured at .895- Suggests that there were patterns to the coding of each paper, meaning that it

is like that we are observing a true construct- Because Inter-rater reliability is so high, this study treats the coders

as effectively identical for purposes of statistical analysis

The Elephant in the Room

What about comparative statistics?- T-tests?- Statistical significance?

The short version: There are no statistically significant differences between conditions on any variable we have tested so far

Case Study: Word Count

How much did word count change from first draft to second?- Looking at all students, across all teachers- Written condition: Mean=120.93, SD=277.64, N=40- Verbal condition: Mean=219.42, SD =290.85, N=50- t stat=1.672, t-critical (2-tail)=1.987- p value (alpha)=0.098- There is an almost 10% chance that the difference in the means is purely

the result of chance

But How Can that Be Chance?

The word count change was almost double in the verbal condition!- But variance (standard deviation) was SUPER high- Essay 3299: 2390 -> 3097 (+707 words added total)- Unusual, but not unheard of for a final portfolio

- Essay 3329: 2971 -> 2235 (-736 words removed total)- Probably an essay that had a lot of unnecessary parts- Removing these words might have indeed been a good idea!

What does this mean?- Sometimes an observed difference can be the product of random

chance- Even a fair coin will have streaks

- There’s almost a 10% chance that the difference in these data is an illusion

What is a Standard Deviation?

A measure of variance- How much data vary- “3, 3, 3, 3” vs “1, 1, 5, 5”

- 2/3s of all samples will fall within 1 standard deviation of the mean- Plus or minus

Why are they important?- A high standard deviation indicates that students are revising in

very different ways

So, What Does This Mean?

Which form of feedback is better?- Right now, there is no clear answer- Student revisions are roughly equivalent- With the high variance in the data, small effects are difficult to

detect- Further research with larger data collection is advised

- Teachers can/should use whatever they like better- Teachers with a preference aren’t clearly disadvantaging students

Do student preferences line up with improved performance?- Numbers of students with a clear preference for verbal (audio)

feedback were simply too rare in this sample to say either way

So, What Does This Mean?

Why are revisions so varied?- Probably because the reasons to revise are so varied- No single kind of revision accounts for - No single pattern of revision can (or should) be seen as “right”

This matches the writing process itself- If the writing process is highly contextual and individualistic,

then the revision process will be similarly contextual and individualistic

But Didn’t You Find Anything?

So far, only 1 statistically significant relationship has been found- Instructor 3’s students exhibited lower word count change in

the audio condition, as compared to the written- Mean audio: 6.17% (N=22, SD= 12.4)- Mean written: 22.2% (N=20, SD= 8.6)

Were there any other effects for Instructor 3’s students?- So far, no- Analysis of Instructor 3’s student revision for type of revision has

not shown any other statistically significant differences- The kinds of revision were similar, but the size of revisions was

smaller

What Does That Mean?

Possibly nothing- Instructor 3’s students may have been outliers- There’s still almost a 3% chance this is not a real effect- A larger data collection would really help

Possibly, it means instructors shouldn’t be forced to use a certain style- Instructor 3 disliked audio feedback- Maybe the students made smaller revisions as a result- Maybe aversion has more of an effect on pedagogy than

preference

What Does This Mean for Future Research?

Revision research requires massive data collection- This study collected 90 essays from 3 instructors, yet samples

were still too small- Probably 400-500 essays is the minimum for a study of this

scope

Institutional support is needed- Coding this quantity of data is very labor intensive- Getting enough instructors and students to participate is

difficult

Overall Finding

Comparative Revision Writing: Quantitative & Qualitative Research on Students’ Perspectives of Multimodal

Instructor Feedback

Ti Macklin – Tialitha.Macklin@csus.eduMark Triana – Mark.Triana@wsu.edu

Stacy Wittstock – Snwittstock@ucdavis.eduWay Jeng – Wjeng@pierce.ctc.edu