USING THE CEFR SCALES TO ASSESS STUDENTS’ PROFICIENCY LEVELS IN A PREPARATORY YEAR CONTEXT OUTSIDE EUROPE

Ebtesam AbdulhaleemPhD Student

Supervised by:Prof. Dr. Claudia Harsch and Dr. Neil Murray

The Study Aim

To explore the potential of self- and tutors’ assessment in determining the proficiency levels of EFL medical and healthcare students in Saudi Preparatory Year Programme (PYP).

Part of a bigger PhD project in which the CEFR is used to perform needs analysis to identify potential gaps.

Overview

Background Information

Concerns

The study purpose

Participants

Methods

Data Analyses

Results

Conclusion

Background Information

Matriculated University Students:

Enrol at the Preparatory Year Programme (PYP)

Placed in three levels (elementary, intermediate and advanced)

Take a high-stakes standardised exit exam at the end of the year.

Join different colleges based on their GPA.

Background Information PYP ELSD Beginner Elementary Pre-

Intermediate

Pre-

Intermediate

Plus

Intermediate Intermediate

Plus

Advanced

CEFR A1 A1-A2 A2 A2-B1 B1 B2 C1

Track Levels

Sem

este

r1

Quarter 1 Quarter 2

Sem

este

r2

Quarter 3 Quarter 4

Me

dic

al/

He

alt

hca

re t

rack Elementary

Elementary

A1-A2

Pre-

Intermediate

Pre-

Intermediate

Plus

Intermediate

B1

Intermediate

Pre-

Intermediate

A2

Pre-

Intermediate

Plus

Intermediate Intermediate

Plus

B1-B2

Advanced Intermediate

B1

Intermediate

Plus

Upper

Intermediate

Advanced

C1

Noticed Concerns:

The students’ results in the final writing exam are of concern

Difficult to have a reliable test that can discriminate the students’ writing proficiency levels.

The Study Purposes:

Identify the students’ writing proficiency levels at the end of PYP as perceived by the students and their tutors

Explore and compare how PYP students and their tutors used the CEFR scales to identify the students’ proficiency levels

By using the CEFR can-do statements, potentially able to gain more detailed insight of students’ actual level of written proficiency, compared to when a standard grade awarded.

From the Literature

To the best of my knowledge, comparing CEFR self- and tutors’ assessment has not been explored empirically in the Saudi context

Using functional language (e.g. can do) increases the accuracy of self-assessment (Ross, 1998)

Advanced students tend to underestimate their levels compared to less proficient students (Sahragard & Mallahi, 2014)

Low proficient students are less accurate in their judgment of their language proficiency (Babaii, Taghaddomi & Pashmforoosh, 2016)

At the end of the academic year 2015

473 PYP students from the medical and healthcare track (All levels)

19 English tutors teaching these students

Participants

Methods

10 selected CEFR scales focusing on writing skills

The students were asked to self-assess themselves

They were given detailed instructions on how to use the scales in addition to an audio-video instructional guide

Meetings with the tutors to explain how the scales should be used

Tutors were asked to assess the same students using the same scales

1 2 3 4 5 6 7 8 9

Ov

erall

writ

ten

pro

du

cti

on

I can write simple isolated phrases and

sentences.

I can write a series of simple phrases and

sentences linked with

simple connectors like ‘and’, ‘but’ and

‘because’.

I can write straightforward

connected texts on a

range of familiar subjects within my field of

interest, by linking a

series of shorter discrete elements into a linear

sequence.

I can write clear, detailed texts on a variety of

subjects related to my

field of interest, synthesising and

evaluating information

and arguments from a number of sources.

I can write clear, well-structured texts of

complex subjects,

underlining the relevant salient issues, expanding

and supporting points of

view at some length with subsidiary points,

reasons and relevant

examples, and rounding off with an appropriate

conclusion.

I can write clear, smoothly flowing,

complex texts in an

appropriate and effective style and a logical

structure which helps the

reader to find significant points.

Ov

erall

writ

ten

in

teracti

on

I can ask for or pass on personal details in

written form.

I can write short, simple formulaic notes relating

to matters in areas of

immediate need.

I can convey information and ideas on abstract as

well as concrete topics,

check information and ask about or explain

problems with

reasonable precision. I can write personal

letters and notes asking

for or conveying simple information of

immediate relevance,

getting across the point I feel to be important.

I can express news and views effectively in

writing, and relate to

those of others.

.

I can express myself in writing with clarity and

precision, relating to the

addressee flexibly and effectively.

As C1

√ √√

√

Data Analyses

Cronbach Alpha

One-way-ANOVA

Students’ self- assessment across PYP levels

Tutors’ assessment (of the same students) across PYP Levels

Independent Sample T-Test

Students’ vs tutors’ Assessment within PYP Levels

Correlation Coefficient Analysis

Weighted Kappa

Results

High reliability was found in students’ self-assessment (α= .884) and tutors’ assessment (α=.951)

The ten scales will be treated as one multi-item scale

Only the average results will be considered for this presentation

Results One-way ANOVA across PYP Levels

F=95.2, P <.001, η 2=0.26 F=113.5, P<.001, η 2 =0.30

Ele

me

nta

ry

Inte

rme

dia

te

Ad

van

ced

Ele

me

nta

ry

Inte

rme

dia

te

Ad

van

ced

Independent Sample T-Test within PYP Levels

M= 4.3, SD=1.5

M= 3.9, SD=1.5

M= 6.6, SD=1.4

M= 5.1, SD=1.6

M= 4.8, SD=1.4

M= 6.59, SD=1.5

t(161)=1.94, P=.05Cohen’s d= 0.26

t(499)= -2.32, P=.02Cohen’s d= -0.14

t(365)=.249, P= .803Cohen’s d= 0.026

Average Scales

Correlation Kappa % Exact

agreement

% within one

adjacent CEFR-

level difference

% within two

adjacent CEFR-

level difference

0.388

P<0.001

0.389 19.0 52.4 79.9

Strength, Direction and Agreement

Proficiency Levels at the End of PYP:

Students’ self-assessment tutors’ assessment PYP Curriculum

Expectations

mean CEFR Levels mean CEFR levels Exit CEFR Levels

Elementary 4.39 B1 3.91 A2+-B1 B1

Intermediate 4.83 B1-B1+ 5.15 B1+ B1-B2

Advanced 6.63 B2-B2+ 6.59 B2-B2+ C1

Conclusion

There is an acceptable agreement between students and tutors which will probably increase with more training (AlFallay, 2004 and Sahragard & Mallahi, 2014)

Using CEFR in a similar controlled way could assist participants in their assessment

Advanced students are more accurate in their assessment compared to lower-level students (Babaii, Taghaddomi & Pashmforoosh, 2016 and Engelhardt & Pfingsthorn, 2013)

Elementary level students tend to overestimate their proficiency levels;

Self and tutor assessment using the CEFR can potentially be used to obtain better insight to the varied aspects of students’ writing proficiency which may be lost through the current practice of exit testing at the PYP.

References AlFallay, I. (2004). The role of some selected psychological and personality traits of the rater

in the accuracy of self-and peer-assessment. System, 32(3), 407-425.

Babaii, E., Taghaddomi, S., & Pashmforoosh, R. (2016). Speaking self-assessment: Mismatches

between learners’ and teachers’ criteria. Language Testing, 33(3), 411-437

Engelhardt, M., & Pfingsthorn, J. (2013). Self-assessment and placement tests–a worthwhile

combination?. Language Learning in Higher Education, 2(1), 75-89.

Ross, S. (1998). Self-assessment in second language testing: A meta-analysis and analysis od experiental factors.” Language Testing 15 (1): 1-20.

Sahragard, R., & Mallahi, O. (2014). Relationship between Iranian EFL learners’ language learning styles, writing proficiency and self-assessment. Procedia-Social and BehavioralSciences, 98, 1611-1620.

Thank you