WASC ARC ConferenceApril 8, 2011, San Francisco, CA
Improving Student Writing and Strengthening Writing Programs – Technology and Techniques That Work
Ellen Junn, Associate Provost
Jennifer Ivie, Assistant Professor of PsychologyKim Morin, Professor of Theatre Arts
William Covino, ProvostCalifornia State University, Fresno
California State University, Fresno (est. 1911)
•23-campus CSU system
•20,932 students
•1,197 faculty
2Copyright © April 2011
Fresno State Demographics
3Copyright © April 2011
Fresno State Demographics
§ First generation college: 68%
§ English proficiency:
63% freshmen require English remediation
4Copyright © April 2011
Multi-pronged Approach to Teaching Writing on our Campus
¨ English Composition ¨ Upper Division Writing Exam
¡ Graduation Requirement¡ Substitution of Designated “W” courses
¨ Writing Requirement for ALL General Education Coursesú Requires “Iterative” writing assignments
5Copyright © April 2011
The Dilemma
¨ High percentage of remedial students. ¨ Perception that surface errors distract
from content. ¨ Effective writing instruction requires
innovative pedagogies.
6Copyright © April 2011
INSTRUCTORS KEEP ASKING…
“Where Is the Time to Respond To All of Those
Essays?”
TECHNOLOGY AS ONE SOLUTION?
•Professor Kim Morin
•“E-scholar” Program
•Upper Division online GE courses
•Faculty interest across campus
A DISCOVERY…Computer Essay Scoring Programs
Several College-level Essay Scoring and Writing programs available…
COMPUTER ESSAY SCORING PROGRAMS
• ETS® Criterion®• http://www.ets.org/criterion/higher_ed/about • IEA Intelligent Essay Assessor
http://www.knowledge-technologies.com/prodIEA.shtml • SAGrader• https://www.sagrader.com/sgm/features• Pearson MyWriting Lab• http://www.mywritinglab.com/whatis.html
ETS CRITERION® SELECTED ON A TRIAL BASIS…
First Trial (Spring 2009): 30 students – 1 essay assignment
First Pilot (Fall 2009)1 Faculty member100 students- 5 essay assignments
About ETS Criterion®• Students write and revise essays
online• Diagnostic Feedback / Holistic Score
within 20 seconds.• Topics Library provides prompts.• Faculty can create topics/prompts. • Errors are highlighted but not
corrected.
12Copyright © April 2011
ABOUT ETS CRITERION®
• No instructor Fees• Simple Registration
• Students purchase Access codes through Bookstore
• Approximately $11.00 per student • One fee provides student use in all classes per
term/semester• Technical Support from ETS®
ABOUT ETS CRITERION®
• Bilingual feedback available • Spanish, Japanese, Simplified
Chinese, Korean, ELL
• Advanced levels available • College–1st & 2nd year, TOEFL, GRE
• No prior essays required
ETS CRITERION® Online Tour
http://www.ets.org/Media/Products/Criterion/tour2/critloader.html
Initial Presentation to Faculty
“Computer Essay Scoring has had a positive effect so far.
I spend more time assessing content, less on grammatical errors.
Students spend more time revising.However, it still misses errors and does not
grade for content.”
16Copyright © April 2011
Early Student Comments
¡ “I like the instant feedback.” ¡ “I like that it goes into depth about why you
didn't get the max score or why you did well.” ¡ “It is easy to use - just copy and paste!”
¡ “I love the fact that I can revise my work for a
better score.” ¡ “It is very helpful and I can see my growth as a
writer.”
Copyright © April 2011 17
Facts About College Student Writing
FACT # 1: Student improvements in writing increase with more
practice ¨ specific, constructive feedback from faculty ¨ opportunity for revisions. IMPLICATION: ü Students need to write or revise more frequentlyü Examine policy requiring “iterative” writing in GE or
W courses
18Copyright © April 2011
FACT # 2: If students do not continue to practice, their writing performance may actually deteriorate. IMPLICATION: ü Examine all writing programs across campusü Identify key courses with writing requirementü Target specific faculty teaching those courses
19Copyright © April 2011
FACT # 3: Teaching students to write effectively can be time-consuming and labor-intensive. IMPLICATION: ü Identify effective technology-related tools ü reduce faculty workloadü provide specific, timely feedback to students.
ü Target part-time faculty who teach writing intensive courses.
20Copyright © April 2011
FACT # 4: Writing instruction involves faculty who are NOT
trained as writing teachers.
IMPLICATION: ü Implement Criterion training for faculty.ü Offer Writing Across the Curriculum workshops.ü Provide professional development funds for all
faculty during Year 1 Pilot. 21Copyright © April 2011
FACT # 5: Assessing & documenting student learning outcomes in
writing performance are key elements to success. IMPLICATION: ü Ongoing data collected for past 3 semestersü Share results with faculty learning community to
determine Best Practices.ü Modify training based on feedback and assessment.
22Copyright © April 2011
Next Came…
ISWI Pilot Launch
Spring 201023Copyright © April 2011
ISWI Improving Student Writing
InitiativeNovember, 2009 - Campus email announcement
calling for faculty participants § Criterion® Training:§ Faculty ISWI Coordinator§ Faculty Learning Community (FLC)
§ Writing Across the Curriculum Workshops§ Assessment of Criterion®
24Copyright © April 2011
Campus-Wide InitiativeBy The Numbers... Year 1 (2 Semesters)
Spring 2010 / Fall 2010
• 349 Classes Involved • 173 Instructors Used Criterion®
• 134 Spring 2010• 68 Fall 2010
• 5,920 Students (Spring 2010)• 3,756 Students (Fall 2010)
Numbers may include duplicates
25Copyright © April 2011
Campus-Wide InitiativeBy The Numbers...
Year 2 (1 Semester) Spring 2011
• 231 Classes Involved • 93 Instructors Used Criterion® • 5442 Students (Spring 2011)• 44,080 Essays Submitted (by March 15)
Numbers may include duplicates
26Copyright © April 2011
Faculty Participation- Year 2
Out of 93 instructors opting to use Criterion, only 20 received Professional Development funds as an incentive in Year 2.
27Copyright © April 2011
Assessments from Spring 2010..
1. Analyzed Criterion® automatically generated data for trends.
2. CLA scores for students with & without
Criterion® 3. First and last papers submitted by students
on Criterion® scored by independent faculty panel
4. Collected student and faculty surveys
28Copyright © April 2011
Automatically-Generated Data
¡ Students who used Criterion® for revision with more than one submission, scored better than those who did not.
¡ Students with a larger number of
assignments and more submissions on Criterion® increased their holistic score on average by approximately 1 level.
29Copyright © April 2011
Collegiate Learning Assessment (CLA)
¡ A significant difference was found between Criterion® users and non-users on the CLA performance task.
¡ No significant differences were found
between the two groups on the analytic writing task.
30Copyright © April 2011
Faculty Scoring Panel
First and last papers submitted by students on Criterion® scored by independent faculty panel.
¨ Overall, student writing improved in
classes that used ETS Criterion®.¡ The greatest improvements were made when
instructors engaged students in substantial discussions of writing and how to use Criterion®.
31Copyright © April 2011
Faculty Feedback – Positive Comments
When using Criterion®, most faculty felt that:1. Papers were easier to grade.2. The program improved their students’
writing skills.3. Students spent more time revising written
assignments.4. Creating assignments was easy. 32Copyright © April 2011
Faculty Feedback - Challenges
Many faculty felt that Criterion®
1. Did not reduce their workload.2. Did not do as much as they had
hoped.3. Was not worth the cost to the
students.
33Copyright © April 2011
Student Survey Response
Copyright © April 2011 34
Student Survey Highlights
What students liked about Criterion®: 1. It was available on-line 24 hours a day. 2. It allowed them to correct grammatical
or mechanical errors before turning in a paper.
3. It gave immediate feedback.
35Copyright © April 2011
What students found challenging:
1. Criterion® identified technical terms or
citations as errors. 2. The program identified errors but did
not correct them. 1. The program did not grade for content.
36Copyright © April 2011
Student Survey Response
Freshmen/sophomores were significantly more likely to agree:
- Criterion has helped me improve my writing.
- Criterion should suggest less and correct more.
- I was able to apply what I learned to other writing assignments.
Copyright © April 2011 37
ESL Student Survey Response
Copyright © April 2011 38
English as a Second Language Student Responses
ESL students were more likely to respond that Criterion® helped them improve their writingúPreferred Criterion® feedback to instructor feedback.úSubmitted their documents more frequently.
- I like the kind of feedback Criterion® gives. - I spent more time improving my writing with Criterion®. - I wish I could use Criterion® for other classes.
39Copyright © April 2011
English as a Second Language Student Response
Copyright © April 2011
http://www.youtube.com/watch?v=pP6feJotLVM
40
Positive Comments From Students
¨ “Awesome program. I find it really useful.”¨ “I like that it is really convenient and I
found myself using the program for all my classes.”
¨ “I started noticing themes in my writing that could use improvement.”
41Copyright © April 2011
Ongoing Research
¨ Studies with Control Groups¨ Continue Collecting Data¨ Determine Best Practices / Uses
Copyright © April 2011 42
Conclusion
So Far, Evidence suggests- ¨ Criterion® helps improve student writing
when combined with effective instructor practice.
¨ Criterion® appears useful for editing grammar and mechanics.
¨ Students who use Criterion® are more engaged with the writing process.
43Copyright © April 2011
Next Steps
¨ Analyze and refine assessment data.¨ Provide online video tutorials.¨ Expand Professional Development &
training.¨ Establish ISWI committee
ú analyze writing instruction across campus.¨ Recognize and thank participating faculty.
44Copyright © April 2011
What Have We Learned? ¨ Students generally found Criterion® to be easy
to use and cost effective.¨ ESL students responded more positively to
Criterion® feedback.¨ Faculty found Criterion® helpful when used to
complement instruction.¨ ISWI shows the high degree of faculty interest
in improving student writing across campus.
45Copyright © April 2011
ISWI Broadly Endorsed & Supported By:¨ Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)¨ Professional Development Subcommittee (subcommittee of Senate’s Personnel
Committee)¨ Chair of the GE Committee¨ Office of Undergraduate Studies¨ Division of Graduate Studies¨ Institutional Research & Assessment Planning Director leading faculty ISWI Assessment
Team¨ Provost’s Office as implemented by the Associate Provost through the Center for the
Scholarly Advancement of Learning and Teaching (CSALT) & Technology Innovations for Learning and Teaching (TILT)
46Copyright © April 2011
Thank You!
¨ William Covino, Provost¨ [email protected]¨ 559-278-2636¨ ¨ Ellen Junn, Associate Provost¨ Interim Director, Center for the
Scholarly Advancement of Learning and Teaching (CSALT)
¨ Interim Senior Academic Technology Officer (SATO) and Technology Innovations for Learning and Teaching (TILT)
¨ [email protected]¨ 559-278-2636
¨ Kim Morin, Professor ¨ Department of Theatre Arts¨ Artistic Director, Theatre for
Young Audiences; English/Drama Credential Advisor
¨ ISWI Faculty Coordinator¨ [email protected] ¨ 559-278-4342¨ ¨ Jennifer Ivie, Assistant Professor ¨ Department of Psychology¨ Interim Director, Center for
the Scholarly Advancement of Learning and Teaching (CSALT)
¨ [email protected] ¨ 559-278-2842
Copyright © April 2011 47
Contact Information- ETS®
ETS® Criterion®¨ http://www.ets.org/criterion
¨ Susan L. Yetman¨ Criterion Account Manager¨ Higher Education, ETS ¨ Rosedale Road MS 51-L¨ Princeton, NJ 08541¨ Direct line: 609.683.2675¨ Toll free: 866.717.1915¨ Fax: 609.683.204
0¨ Email: [email protected]
Copyright © April 2011 48
Arthur RuzzanoWestern Regional ManagerHigher Education Assessment SolutionsPhone: 310.944.4034Fax: 609.683.2040Email: [email protected] Bill WynneProduct Manager - ETS Proficiency ProfileETS Programs and Services DivisionTelephone (609) 683-2006E-mail [email protected]