Download ppt - Changing Tests for Teachers: English Language School-Based Assessment in Hong Kong Chris Davison Faculty of Education, University of Hong Kong University

Changing Tests for Teachers: English

Language School-Based Assessment in Hong Kong

Chris DavisonFaculty of Education, University of Hong

Kong

University of Hong Kong Faculty of Education

Inaugural Seminar Series

English Language School-Based Assessment: Integrating Theory and Practice, Dec 19th 2005

Curriculum and assessment reform in Hong Kong The Hong Kong Examinations and

Assessment Authority (HKEAA) has recently moved from norm-referenced to standards-referenced assessment, including the incorporation of a substantial school-based summative oral assessment component into the compulsory English language subject in the Hong Kong Certificate of Education Examination (HKCEE).

Curriculum and assessment reform in Hong KongThe initiative aims to align assessment more closely with

current English Language teaching syllabus (Curriculum Development Council, 1999) and the new outcomes-based Senior Secondary curriculum.

to assess learners’ achievement in areas not easily assessed by public examinations, in particular speaking and extensive reading.

enhance student self-evaluation and life-long learning.

The SBA initiative

Starting in S.4, teachers assess their own students’ oral English language competencies through a range of authentic classroom-embedded activities over 2 years.

Initiative developed by a team of researchers at the Faculty of Education, University of Hong Kong, in collaboration with HKEAA.

Part of an evolving but coherent and systematic programme of research and evaluation, professional development and system-wide support

The SBA initiative SBA is integrated into the teaching and

learning process, with teachers involved at all stages of the assessment cycle: planning the assessment programme identifying and developing appropriate assessment tasks overseeing the assessment process

making final judgments

The SBA initiative But previous studies have found … Changes to summative assessment did not

automatically lead to improvement in learning (Andrews 1994; Cheng 1998; Andrews, Fullilove & Wong 2002).

Assessment innovation was severely constrained by traditional school culture and by teacher, parental and student expectations (Cheung & Ng 2000; Carless 2001; Adamson & Davison 2003).

Wide variation in teachers’ interpretations of student performance and of their role in the assessment process (Yung 2001).

The SBA content and structure

Aims Content Structure Assessment criteria Making judgments Standardization Safeguards

SBA as research The SBA initiative has stimulated multi-level and multi-purpose research efforts from

a growing research team with more than 12 interrelated research projects now underway, or under review.

In addition, the research efforts have attracted the interest of a growing number of MPhil and PhD students, as well as the attention of the international assessment and English language education research community.

SBA as research

The particular characteristics of this research that makes it so interesting: its significance to the educational community its high-impact and visibility its concern to address both fundamental theoretical problems in language assessment (basic

research) and the needs of school communities (applied research) its action-oriented collaborative approach

SBA as research Many concerns and issues about SBA have been systematically gathered from key

stakeholder groups during the process of SBA development, implementation and evaluation.

Data was gathered over a year via questionnaires, individual and focus interviews, classroom observation and stimulated recall, and public briefing sessions as well as from the trialling of the tasks and assessment processes, and the training workshops, in total involving more than 1800 teachers and over 600 schools.

SBA as research

Adapting Brindley’s (1995) taxonomy, these

concerns and issues can be divided into three

types: Sociocultural (political) Theoretical (technical) Practical

Socio-cultural issuesStill a traditional testing culture in Hong Kong: Assessment practices are still primarily

oriented towards providing data to select students for education or employment (Biggs 1995).

External exam results are still the dominant way schools, students and teachers are evaluated and held accountable.

Exam preparation is the traditional role of assessment in the senior secondary classroom (cf. classroom-based assessment).

Socio-cultural issues Traditional cultural practices exemplified in

a study of writing assessment in senior secondary English (Davison & Tang, 2003):“Many teachers reported they cannot ‘assess’, only ‘mark’. They feel unable to make a difference in teaching and learning, to respond to individual needs, because of community expectations of convergence and commonality. Teachers feel their assessment process are expected to change, without fundamental purposes being explicitly challenged. Such role conflict results in increasing stress and a decline in perceived teacher expertise.”

Socio-cultural issues

However, the official adoption of the UK Assessment Reform Group’s (1998) distinction between assessment for learning, and assessment of learning has stimulated the beginnings of a major paradigm shift

Hong Kong schools are moving from a culture of testing to a learning and assessment culture (Hamp- Lyons 1999; in press).


Some teachers in the SBA study perceived cultural differences as a major stumbling block to assessment reform:“I feel it takes time because the culture. The education culture in Hong Kong is different from other western countries and the students may not used to that kind of assessment. They like to do exam paper, they think they have something to learn.”


Another teacher newly returned from overseas, commented:“Actually I was so surprised that ... How slow we Hong Kong people are in terms of education… because I remembered when I was in Canada, we never ... You would never be graded on just one exam. It’s quite like what we are trying to do actually, I believe that (assessment for learning) has been practised in those places for years and I was actually surprised nobody did anything (here)… I am totally for assessment for learning.”

Socio-cultural issuesResearch shows need for much more understanding, even at the system level, of …

the dual role of the teacher as facilitator and assessor

the concept of a student being assessed against criteria (rather than other students)

the need for teacher consultation and interaction as part of the standardization process.

Socio-cultural issues The schools which were already doing

extensive reading and whose students engaged in oral group work and individual presentations on a regular basis found it the easiest to integrate the assessment tasks into their existing practice:“I just briefly tell the students about the task because it is in mid May, so they were quite busy that moment. So I asked them just make use of what they have been doing, say they just, they can just took from ERS report and work on it, say prepare a better review so that they can just have their presentation based on the review and I told them …

Socio-cultural issues …That’s what I did at the very beginning. Later on,

I, I met them some days later, and I asked them to show me the book review they had written and I took a look at it and I found that there weren’t any major problems in it. So I just returned them the review and they started to prepare those tasks … and later on, just right before they did the presentation, I helped them with the vocabulary and the names because they didn’t know how to pronounce them. So I just helped them pronounce them correctly… (but) I gave them more guidance according to the SBA documents … because the five questions listed there (see Appendix II) suggested some sort of high order thinking skills … So I try to scaffold them to think in that way.”

Socio-cultural issues Some teachers took longer to come to grips

with the principles involved in SBA, and their implications for teaching and learning as well as for assessment practice:“For students of higher forms, the time (8 min) is quite limited. They can’t have enough time to introduce their books and ask each other questions.”

… but no time limits actually prescribed in SBA.


‘Fairness’ was – and is - a major concern. Fairness is fundamentally a sociocultural, rather

than a technical, issue, “a justice that goes beyond acting in agreed upon ways and seeks to look at justice of the arrangements leading up to and resulting from those actions.” (Stobart, 2005, p. 1).

“Fair assessment cannot be considered in isolation from both the curriculum and the educational opportunities of the students … some similarities to the more traditional notion of ‘fairness’ embodied in the classical examinations for the Chinese civil service—the concept that conditions should be consciously created to make opportunities open to all” (Hamp-Lyons, 2005).

Socio-cultural issuesFairness in the current HKU examination system

Fairness in the SBA component

Treat all students equally(vs. equitably)

Treat each student individually (i.e. unequally)

Same task, same input, same conditions, same length of time for all

Different arrangements for different students according to their language proficiency

Major aim: To give a consistent grade to each student in an invariable settings for selection purposes

Major aim:To provide opportunity for all students to demonstrate their best. Learning culture for all sense of achievement.

Socio-cultural issues Preliminary findings suggest that these

different approaches to fairness can be reconciled to some extent by providing teachers with opportunity to tailor classroom-based assessments to the needs of their students, according to commonly-agreed processes, outcomes and standards, with teacher assumptions about students and their oral language levels being made explicit through collaborative sharing and discussion with other teachers.

Theoretical issuesAt the theoretical level concerns with SBA

revolve around the understanding and interpretation of traditional concepts such as reliability, validity and authenticity, eg.

The students will memorize everything. Some teachers will tell students the task ahead of

time Teachers will favour their best students. Panel chairs will make all teacher agree with

them. Schools will make up results. Tutorial schools will coach students. The HKEAA should take up all the ‘scripts’ to

check the accuracy of the marks.

Theoretical issues

“The traditional positivist position on language testing, with the tendency to map the standard psychometric criteria of reliability and validity onto the classroom assessment procedures, has been called into question, and the scope of validity has been significantly broadened (Chapelle 1999; Lynch 2001, 2003; McNamara 2001) and taken further by a number of researchers.” (Rea-Dickins, in press)

Theoretical issues

Defining characteristics of school-based assessment (Stiggins & Conklin 1992; Black & Wiliam 1998; Brookhart 2003):

Teacher-mediated Co-constructed and dialogic Context-dependent Multiple and varied Dynamic and evolving

Theoretical issues“Such an approach can be seen as constructive and enabling because of its focus on assessing the process of learning, its attempt to elicit elaborated performance, and its emphasis on collaborative activity …”

Rea-Dickins (in press)

Theoretical issues

However, if there is no reinterpretation of traditional conceptualizations of reliability and validity, SBA may be reduced to

a series of summative mini-achievement tests external to the teaching and learning programme.

the assessment of rehearsed monologues or dialogues with little or no opportunity for authentic language use.

competition rather than collaboration

Theoretical issues

Cf. Clapham (2000, p. 21)Traditional test criteria do apply to alternative assessment:“A problem with methods of alternative assessment, however, lies with their validity and reliability: Tasks are often not tried out to see whether they produce the desired linguistic information; marking criteria are not investigated to see whether they ‘work’; and raters are often not trained to give consistent marks.”

Theoretical issuesPreliminary findings suggest … Potentially much greater validity

with SBA than external oral paper - More natural and authentic tasks

- Students have a genuine reason to communicate as have read different texts- Students in comfortable, familiar environment-- Class teacher already familiar with the range of student performance, can ask questions to ensure the text is student’s own ‘work’ little possibility of cheating

Theoretical issues Potentially much greater reliability with

SBA than external oral paper - A series of assessments (instead of one) by a teacher who is familiar with the students

- Exemplars of student ‘performances’ can provide a strong basis for discussing and internalizing the set standards.- Multiple opportunities for assessor reflection and standardization.- Disagreement between teachers the key to establishing trust (and trustworthiness).

Theoretical issues

But initially lots of misconceptions … Superficial tasks A lot of over-rehearsal and memorization Use of buzzers and rigid timelines Over-reliance on scripts and formula An emphasis on performance and

recording Overbearing or stressed teachers Unhappy students

Practical issues Not surprisingly, many practical concerns:

- Access to appropriate assessment and extensive reading) resources- Access to activities and techniques, models/resources- Access to technical resources/expertise - Lack of recognition/support at the school-level- Concerns about student and parental expectations- (In)adequacy of training- Lack of time- Competing priorities

Practical issues Teachers also expressed a need to

understand much better the underlying assumptions of SBA, how to modify their teaching and learning, and how to set up effective assessment tasks:“Many teachers have an urgent need to view others’ practices and share experiences ... We can film the good lessons for teachers and analyze the lessons. We (need to) focus on teaching instead of assessment only.”

Practical issues

More training will be provided to all teachers teaching S4 English between January and April so that they can be supported during the whole of the assessment period in second semester. SBA handbook, introductory DVD and booklet are available, on-line and face to face support.

Practical issuesPreliminary findings suggest … Much extra time and effort required during

the first year(s) of implementation as SBA is still new to both teachers and students, but once SBA a routine part of classroom activities, should be no significant increase in student/teacher workload?

SBA changes nature of what is done rather than how much is done; the key is to integrate assessment, teaching and learning?

Positive impact on teachers, learners and schools

Despite many concerns, the attitudes towards SBA by teachers, students and school communities are generally very positive:“Personally, I enjoyed this trialing experience. I learnt how to judge the students through this activity. Moreover, my students tried to do the presentation based on the guiding questions given to them. Students found this presentation quite interesting and motivating. They learnt how to speak confidently and bravely during this assessment activity. They found this presentation rewarding since they can learn not only from the book but also through their actual participating experience.”


Those teachers involved now much more aware of values and principles underlying SBA, and options now available to them:“After my students … had finished the presentation, I give them feedback and try to improve their performance and … I was really amazed by the response … I thought that one of my students actually is not very good in English, but after the presentation she tried very hard to do it again and again. Feedback really works and I found that my students have improved a lot.”


Teachers commented on increase in student confidence and positive effect of assessment activity on other language skills:M: I think my student Sandy, just the one, just videotaped. She has great improvement… She thinks that is useful, very useful and she told me that … it also helps her reading. I think the student can really improve a lot after this trialing.

C: Well, my student has shown a great improvement in terms of confidence and English proficiency. They like talking to each other in English and they are not being afraid of being videotaped.


A:Yeah, they start, they start reading English books as well.S: I think is just like what you said, is very good training for confidence and the students actually articulate what they read and … they found reading very useful, purposeful, that is something that you can share with someone, is not something just happening in your inner self. So I think the one that I trained out of the two, one girl and she does quite well even in her writing. So she did well in this writing exam. I’m not sure if it is the effect of that training experience (but) it seems that she gained some confidence during this period of time.


Students also commented favorably on the assessment activities:T: What do you think about the assessment task you did in the presentation task 5?S1: It was quite interesting that we need to think about what the character needs. We can buy a gift.S2: I just think it’s easy to handle it.T: Why?S2: It’s quite interesting to think for a gift to the character. T: When you are thinking of a gift is it difficult?


S2: I don’t think so because I can think of many, many gifts to solve the problem.T: Did you enjoy working with your partners?S2: Yes. I did because my partners are all my best friends. We didn’t have any gaps so we did the project perfect.T: What about you?S3: I also very enjoy doing the task with my friends as they know me very much. When I don’t know what can I say they will help me to continue the conversation.S4: With the partner I won’t feel nervous.


Students can comment on own development and receive constructive feedback immediately after assessment, hence improving learning.

Students encouraged to work consistently. Teachers and students become partners in the

assessment process Collaboration and sharing of expertise take

place within and across schools. Teachers build knowledge and skills in

alternative assessment, readily transferable to other areas of English language curriculum, and beyond.

Conclusions

The SBA initiative, is a major educational reform, entailing significant changes in school culture and structures as well as in pedagogic expectations among students, teachers, administrators, parents and the wider community.

The SBA initiative requires the development of content- and context-appropriate assessment activities, instruments and procedures which are explicitly linked to high-quality teaching and learning.

Conclusions The SBA initiative requires English language

teachers who are not only confident and skilled at making highly-contextualised, consistent and trustworthy assessment decisions, but also effective at involving students in the assessment process.

The SBA initiative requires high levels of teacher collaboration, leadership and support within and across schools.

SO … much research and (informed) practice is needed to help make this initiative a success.

Assessment for learning in Hong Konghttp://cd.emb.gov.hk/basic_guide/BEGuideeng0821/chapter05.html

“Based on the beliefs that every student is unique and possesses the ability to learn, and that we should develop their multiple intelligences and potentials … there should be a change in assessment practices and schools should put more emphasis on 'Assessment for Learning' as an integral part of the learning, teaching and assessment cycle …

… In other words, teachers should use assessments (e.g. as simple as effective verbal questioning, observation of student behaviour) and provide immediate feedback to enhance student learning in everyday classroom lessons. The focus is on why they do not learn well and how to help them to improve rather than just to use assessments to find out what knowledge students have learned”

Selected References

Adamson, B. & Davison, C. Innovation in English language teaching in Hong Kong primary schools: One step forwards, two steps sideways. Prospect, 18 (1), 27-41.

Andrews, S. 1994. The washback effect of examinations: Its impact upon curriculum innovation in English language teaching. Curriculum Forum, 4 Carless, D., (2005). Prospects for the implementation of assessment for learning. Assessment in Education, 12(1), 39-54.

Chapelle, C. (1999). Validity in language assessment. Annual Review of Applied Linguistics, 19, 254-272.

Cheng, L.Y. 1998. Does washback influence teaching? Implications for Hong Kong. Language and Education 11 (1), 38-54.

Selected References

Davison, C. & Tang, R. (2003, March). Assessing in the swamp: Formative assessment in Hong Kong secondary school English. Paper presented at the American Association for Applied Linguistics, Arlington, VA, USA.

Hamp-Lyons, L. (in press). The impact of testing practices on teaching: Ideologies and alternatives. Classroom-based assessment: Possibilities and pitfalls. In Cummins, J. & Davison, C. (Eds) The International Handbook of English language teaching, (Vol. 1), Norwell, MA: Springer.

Lynch, B. (2003). Language assessment and programme evaluation. Edinburgh: Edinburgh University Press.

McNamara, T. (2001). Language assessment as social practice: challenges for research. Language Testing, 18(4), 333-349.

Rea-Dickins, P. (in press). Classroom-based assessment: Possibilities and pitfalls. In Cummins, J. & Davison, C. (Eds) The International Handbook of English language teaching, (Vol. 1), Norwell, MA: Springer.

SBA Consultancy Team (2005). 2007 HKCE English examination: Introduction to the school –based assessment component. Hong Kong: HKEAA.