Implementing Exit Exams in a University-level Intensive ESL Program

Implementing Exit Exams in a

University-level Intensive ESL Program

Breana BayraktarMarietta BradinovaMasha VassilievaVirginia International University

Plan•Situate this project in the context of an

intensive, university-level ESL program•Describe the creation and implementation

of an alternative essay rating guide•Share tentative conclusions and discuss

implications for other university ESL programs.

March 18, 2011Presentation at the 2011 TESOL National Convention

Background on the program•University-level ESL program serving

students from all over the world•4 levels of instruction:

▫Elementary▫Intermediate▫Advanced▫High-Advanced (also known as College Prep)

•Completion of the college prep level = advancement to academic programs, “General Education” courses, at the undergraduate and graduate level.


Overview of Placement/Advancement

• Placement of all new students▫Computer-based test (ACCUPLACER)

Reading Skills evaluates comprehension of short passages

Language Use measures grammar and usage Sentence Meaning assesses the understanding of

word meanings in one- or two-sentence contexts ▫Essay (45 mins)▫Interview

• Advancement of all continuing students▫Pass 80% of their classes at the previous level with

an average of 75% or higher. ▫Or, re-take the placement test to see if they are

eligible to advance to the next level.March 18, 2011Presentation at the 2011 TESOL National Convention

Concerns regarding placement/advancement procedures•Students can “pass” and graduate from the

program by passing a majority of their classes at each level▫program advancement protocol allows student to

advance if they pass 80% their courses, or if they can perform well enough on the exit test to prove that they have “tested into” the next level.

•Anecdotal evidence from instructors ▫wide range of proficiency levels in each class▫students demonstrate an inadequate mastery of

grammar and sentence structure, particularly at the higher levels


Our Puzzlement• Concern that the essay scoring rubric didn’t give

enough weight to grammatical accuracy and the descriptors were too vague.

• Feeling that movement at the CP level—both into the level and advancing out of the level—was possible for students lacking important grammar skills.▫Did the course grades (for continuing students) or the

placement essay rubric not give sufficient weight to grammatical accuracy?

• Our goal: to create and test an essay-rating mechanism specifically for students entering and exiting the college prep level that would:▫Give grammar significant consideration ▫Produce consistent ratings among raters


Our thinking behind revising the rubric•Concern that the current rubric (side 1 of

the handout) wasn’t precise enough for the upper levels, particularly the placement in college prep vs. academic courses

•Desire for rubric to reflect what courses at the ESL level actually focus on, so that students are placed appropriately

•Need for greater emphasis on mastery of language functions (as opposed to rhetorical functions)


New Essay Rating Diagram

G1

G2

G3

OS1

OS2

OS3

OS4

Academic (non-ESL) level

College Prep level

College Prep level

Advanced level

OS3

OS4

Advanced level

Intermediate level


Grammar Descriptors (G1) Grammar errors are either nonexistent or barely noticeable / minor. They do not distract from the overall essay; nor do they impede comprehension or slow the reader down. Students moving on to the Academic level need to be able to write clearly and comprehensibly, and be prepared to succeed in a “freshman composition” class, where the strategies of academic writing will be taught, but where there is not a focus on grammar.


The first descriptor (G1) for GRAMMAR has the following options for ORGANIZATION/STRUCTURE:

G1

(OS1) There is a clear main idea with a controlling idea and a conclusion or a concluding remark. The body is organized logically (not necessarily into paragraphs) and there is evidence of sequencing & connectors. For each major idea there is support (examples, details, explanations).(OS2) There is a main idea but no clear controlling idea. There is a lack of logical organization. There is insufficient or irrelevant support in the body. There are evident limitations of vocabulary in the form of non-specific vocabulary ("things" or "stuff") and/or not-appropriate vocabulary. Either the lack of organization or vocabulary limitations are distracting to the point of slowing the reader down.

Placement in Academic (non-ESL) level

Placement in College Prep level


Grammar Descriptors (G2) Grammar errors are noticeable, but significant portions (chunks of 3-4 sentences) show mastery of basic grammatical forms. There may be consistent errors in the choice of prepositions and/or articles. There is some evidence of the ability to use complex language accurately, although not necessarily consistently, such as advanced verb tense forms, subordinate clauses, gerunds, modifiers, etc.

Students at the College Prep level need to be able to consistently use basic grammatical forms accurately, and be ready to receive support in developing more advanced structures within the context of learning to write academic essays.


The second descriptor (G2) for GRAMMAR has the following options for ORGANIZATION/STRUCTURE:

G2

(OS3) The writing is of a sufficient amount (i.e., a longer paragraph) to allow the reader to identify one or more main points. There is some attempt at organization and support or development. (OS4) The writing is a short paragraph or collection of sentences. There is no evidence of logical organization. There is irrelevant information and/or the writing may not address a clear topic.

Placement in College Prep level

Placement in Advanced level


Grammar Descriptors

(G3) Most sentences have grammatical errors of different types. There are frequent errors even with basic forms and with word order.

Students at the Advanced level need to be ready to receive support in developing accuracy in the use of grammatical structures within the context of learning to write longer and more complex paragraphs.


The third descriptor (G3) for GRAMMAR has the following options for ORGANIZATION/STRUCTURE:

G3

(OS3) The writing is of a sufficient amount (i.e., a longer paragraph) to allow the reader to identify one or more main points. There is some attempt at organization and support or development.

(OS4) The writing is a short paragraph or collection of sentences. There is no evidence of logical organization. There is irrelevant information and/or the writing may not address a clear topic.

Placement in Advanced level

Placement in Intermediate level


Data examined•4 semesters of college prep students

(n=75)•Entrance and exit tests

▫Computer-based test ▫Essay


Improvement in exit test scoresFall 2009 Spring

2010Summer

2010 Fall 2010 TOTAL

Average Gain on Computer Test (one level spans 45

points)

21.39(approx. ½ level gain)

28.17(a little over ½

level gain)

40.00(a little less

than 1 level gain)

28.34(a little over ½

level gain)

34.26(a little over ¾

level gain)

Average Gain on OLD Essay

(one level spans 1 point)0.34 0.81 0.53 0.21 0.47

Average Gain on NEW Essay

(one level spans 1 point)0.74 0.53 0.13 0.71 0.56


Using the new rubric...•Rescored entrance essays:

▫7 students scored lower than college prep on their entrance essays (score = or < 3.5)

▫10 scored at college prep level (score > 3.5)

▫5 scored at academic level (score > 4.5)•Rescored exit essays:

▫2 scored lower than college prep ▫13 scored at college prep level ▫7 scored at academic level


Can we score the new essay reliably?

•20 essays (of 44 total) had unanimous scoring across all scores (between 3-5 scorers)

•All the rest had 3 of 4 scorers in agreement, with 3 exceptions

•For the 3 essays with differences of more than 1 level (i.e., one scorer assigned a 5, another a 4, and another a 3), in each case the essay was particularly difficult to read (bad photocopy).


Some tentative conclusions...• Average gains of approx. ½ level on all measures

(ACCUPLACER, Old Essay & New Essay) tentatively indicate that program expectations that students are ready to move up to Academic level work after 1 semester of College Prep are problematic.

• New scoring system▫Fast and easy to use, and also looks promising in

terms of reliability between program faculty.▫Where there was disagreement in scoring using the

new essay scoring system, it was largely at the grammar level—indicates that further consideration of the grammar level descriptors is needed.


Recommendations we are consideringShort-term…• Adding an optional fifth level which will be a

combination of General Education courses and ESL courses (GED: Academic Writing, GED: Oral Communication Skills, ESL: Reading, ESL: Academic Listening & Note Taking).

Long-term…• Repeat this study with a more “traditional” essay rubric

to see what results we get.• Repeat this study with students at the other three levels

to see how their exit scores compare with their initial placement results. ▫ Are we expecting them to make unrealistic progress over

the course of 15 weeks?▫ Should we restructure the program, and splitting the

current 4-level (4-semester) curriculum into five or six semesters?


Implications• Importance of multiple measures (computer test

scores, essay scores, interview, previous course grades)

• Importance of faculty/instructor input in the placement/advancement process▫“highly recommend that multiple measures be

employed in any EAP placement process and that regardless of the tools employed, faculty be involved in interpreting the results” (James & Templeman, 2009).

• Importance of gathering assessment data to track student progress and for program improvement


Selected ReferencesJames, C. & Templeman, E. (2009, Spring). A Case for Faculty

Involvement in EAP Placement Testing. TESLCanada Journal/Revue TESL Du Canada 26(2), pp.82-99.

Brunk-Chavez, B. & Fredericksen, E. (Fall/Winter 2008). Predicting Success: Increasing Retention and Pass Rates in College Composition. Writing Program Administration, 32(1), pp.76-96.

Andrade, H.G. (Dec. 2005). Understanding Rubrics. Learnweb. http://learnweb.harvard.edu/alps/thinking/docs/rubricar.htm

Andrade, H.G. (2000). Using Rubrics to Promote Thinking and Learning. Educational Leadership 57(5), pp.13–18.

Wilson, M. (2006). Rethinking Rubrics in Writing Assessment. Portsmouth: Heinemann.


Questions?Please contact us with comments,

suggestions, questions…

Breana [email protected]

•This presentation will be posted at:http://breanabayraktar.wordpress.com


Documents

Implementing Exit Exams in a University-level Intensive ESL Program