Impact of Prior Experience (Sample)

8/13/2019 Impact of Prior Experience (Sample)

1/97

TitleThe impact of prior experience on teachers' perceptionsof standards-referenced assessment

Author(s) Ng, Ka-ming;

Citation

Issue Date 2007

URL http://hdl.handle.net/10722/51294

RightsThe author retains all proprietary rights, (such as patentrights) and the right to use in future works.


2/97

The impact of prior experience on teachers perceptions of

standards-referenced assessment

By

NG Ka-ming

(U. No.: 1991160744)

August 2007


3/97

i

Acknowledgments

I am most grateful to my academic advisor, Dr. Philip Stimpson. His

professional guidance and continual support is of vital importance to my research.

Indeed, I have learnt a lot from him regarding research, teaching and other personal

matters. I would like to express my sincere gratitude to Mr. Gwyn Edward, Dr. Pang

Ming Fai and Prof. Mark Bray who served on my coursework. What I learned from

them over the years would fill many more pages than this dissertation. Also thanks to

Ms Gigi Lui for her constructive suggestions and kind words of encouragement.


4/97

ii

Declaration

I hereby declare that this dissertation represents my own work and that it has

not been previously submitted to this University or any other institution in application

for admission to a degree, diploma or other qualifications.

_______________________

August 30, 2007


5/97

iii

Abstract

Whilst many studies investigating teachers concern have been carried out to foster

curriculum change in the past, this paper examines teachers views of the recent

assessment reform, the adoption of standards-referenced approach, in an attempt to

ascertain what concerns appear prevalent as the hindrance to the change. By using

Concern Based Adoption Model 253 teachers stages of concern are collected and

analysed, and five interviews are conducted to consolidate the current beliefs about

standards-referenced assessment in practice. This study finds that the hindrances to

the educational change are the poor quality of teacher support, the obstructive

environment for the dissemination of information about SRA, the untimely

implementation, perceived heavy workload, and stultifying school management and

policy. Finally, this paper explores the good practice in educational change.


6/97

iv

Contents

Acknowledgements .................................................................................. i

Declaration ............................................................................................ ii

Abstract .................................................................................................. iii

Table of Contents .................................................................................... iv

Chapter 1: Introduction

1.1 Background .................................................................................... 1

1.2 Adoption of standards-referenced assessment by teachers .............. 4

1.3 Teachers concerns about the innovation of SRA ............................ 6

1.4 The aim of the study ....................................................................... 8

1.5 Research questions ......................................................................... 8

1.6 The structure of the dissertation ...................................................... 9

Chapter 2: Literature Review

2.1 Introduction ................................................................................... 10

2.2 Standards Referencing .................................................................... 10

2.3 Curriculum Change ........................................................................ 19

2.4 Teachers perception and Concerns Based Adoption Model ............ 22

2.5 Conclusion ..................................................................................... 25

Chapter 3: Methodology

3.1 Introduction ................................................................................... 26

3.2 CBAM approach ............................................................................ 27

3.3 Research design ............................................................................. 28

3.4 The operationalisation of Stages of Concern Questionnaire .............. 31

3.5 Follow-up interviews ..................................................................... 37


7/97

v

Chapter 4: Findings

4.1 Introduction ................................................................................... 42

4.2 Demographics of the study ............................................................. 42

4.3 Stages of Concern derived from the questionnaire survey ............... 44

4.4 The interview findings ................................................................... 47

4.5 The impact of prior experience ....................................................... 55

4.6 The hindrances to the change ......................................................... 60

4.7 Conclusion ..................................................................................... 64

Chapter 5: Conclusions

5.1 Introduction ................................................................................... 65

5.2 Concerns of teachers about the innovation (Research Question 1) .. 65

5.3 Factors influencing variation in concern (Research Question 2) ...... 66

5.4 Hindrances to the change (Research Question 3) ............................ 69

5.5 Limitations ..................................................................................... 71

5.6 Implications of findings for practice ............................................... 74

5.7 Implications for research ................................................................ 78

5.8 Concluding Remarks ...................................................................... 79

Appendix A (Stages of concern questionnaire) .................................... 80

References ............................................................................................. 83


8/97

1

CHAPTER 1

Introduction

1.1 Background

The Hong Kong public assessment system is undergoing fundamental conceptual reform

including a move to standards referencing in which students are assessed against stated

standards or criteria. In the current school leaving public examinations at 17+ and 19+

years, the Hong Kong Certificate of Education Examination (HKCEE) and Hong Kong

Advanced Level Examination (HKALE), a norm-referenced grading methodology is

adopted for the majority of subjects (Choi, 1999) but this is to be changed to a

standards-referenced system in 2009 (EMB, 2005). This has followed views expressed in

1993 by the Education and Manpower Branch of the Hong Kong Special Administrative

Region (HKSAR) government that norm-referencing is inappropriate in a school system

to be introduced dedicated to the development of individual potential as opposed to

selecting an elite minority (Biggs, 2003, p.173). This study is concerned with teachers

perceptions of the change.

Under the current grading system within the two public examinations, the

norm-referenced design is such that only a set proportion of candidates can obtain a high

grade; a proportion must also fail although these proportions may vary as the examination

paper levels of difficulty vary slightly from year to year. For example, in the 2006

HKALE, Use of English was the most numerically popular subject, but only 13.8% of the

candidature were able to obtain a C grade or above within a system of six coarse grades

(A, B, C, D, E and F) (HKEAA, 2006) and 0.6% of the candidature obtained an A grade.

This norm-referenced approach, which the Hong Kong Examinations and Assessment

Authority (HKEAA) uses, more or less yields similar figures across the years. This


9/97

2

situation, in which a large portion of candidates obtain lower grades as having achieved

nothing, was accepted by the Hong Kong society because the HKALE largely served

selection (Choi, 1999). This was acceptable in an environment where opportunities were

strongly limited. This is less so now. Before 1986, less than 4% of the 17-20 age cohort

enrolled tertiary institutions in Hong Kong (UGC, 2005b) and the HKALE, at that time

offered a sufficient platform for universities to select elite students. However, throughout

the 1980s, there was a growth of tertiary institutions and first-degree places available as

the University Grant Committee (UGC) (the University and Polytechnic Grants

Committee during 1980s) supported increased funding for higher education. Growth in

the 1990s was such that the Government intended to reach a tertiary participation rate of

18% with 14,500 (UGC, 1993). Thus, by 2004-05, there were 14,585 first-year students

in UGC-funded tertiary institutions (UGC, 2005a) representing a four-fold increase over

the previous 20 years. Almost 50% of the HKALE candidates found a place in a tertiary

institution in 2005. The need for a highly selective examination design became less

crucial than the situation two or three decades earlier.

If a norm-referenced system was now less required for picking out elites, it was

equally inappropriate for the low achievers. Every year, there are around 6,000 candidates

of the HKCEE obtaining a zero grade point, which means they have no subjects at E

grade or above. They are labelled as zero-mark students. The existing grading

methodology limits the grade distribution in each subject and cannot be changed

significantly even though the entire candidature performs well overall in the examination.

This system is discouraging many students who may go away with the misperception that

they have achieved nothing. Zero-mark does not mean achieving nothing at all. In fact,

they have something but at a lower level than the others.

The HKEAA initiated a study called the Review of Public Examination System


10/97

3

(ROPES) in 1998. The ROPES (1998) noted that public examinations no longer serve

merely the need of an elitist cohort. In 2003, the Review of the Academic Structure for

Senior Secondary Education and Interface with Higher Education (RASIH) report

(Education Commission, 2003) concluded that the existing grading mechanism of public

examinations falls short of meeting the current expectations of community. The function

of the examinations has gradually moved from selection to certification. At a similar time,

the Education Commission in Hong Kong, an advisory body for the Government,

proposed general education reforms and suggested new mechanisms for assessing student

performance that embraced three functions: facilitating teaching and learning,

certification, and selection. It further emphasised that public examinations needed to

serve certification as well as selection. Therefore, reform of public examinations was

proposed and subsequently adopted (EMB, 2005). The reform proposal suggested the

adoption of a standards-referenced approach in public examinations arguing that this

would improve the grading mechanism.

A standards-referenced approach became a blueprint for assessment reform and a

new single qualification at Senior Secondary Education level, the Hong Kong Diploma of

Secondary Education (HKDSE), combines the HKCEE and the HKALE. The new system

would adopt a standards-referenced approach for reporting results. It would involve five

descriptor levels for each standard although to placate universities, Level 5, the highest,

was to adopt a norm-referenced approach in which Level 5** and Level 5* represent the

top 1% and the next 3% of the candidature (EMB, 2005, pp. 9-10). The HKDSE is thus

not a purely standards-referenced examination as in some other examination boards.

Patently, it is a hybrid approach. Nevertheless, feedback from public consultations reveals

that this approach is not rejected but not always fully accepted by teachers and other

stakeholders.


11/97

4

1.2 Adoption of Standards-Referenced Assessment by Teachers

Hong Kong has been undergoing a raft of education reforms in decades. Most education

reforms in Hong Kong were brought in because of political and economical issues and

had substantially ameliorated the education system (Sweeting & Morris, 1993). Some

education reforms delineated the influence of the handover to Chinese sovereignty in

1997. The reforms in the transitional period to Chinese sovereignty (1984-1997) were

introduced at a rapid rate and seemed to be symbolic. The post-handover government has

inherited the problems in the education system and the symbolic reforms; the subsequent

reforms are then forceful (Morris & Scott, 2003). Some reforms (for example, nine-year

free and compulsory education) are successful, but some reforms such as Target-Oriented

Curriculum have encountered many encumbrances that many palliatives are embodied in

the implementation (Lam, 2003).

Reform means to make an improvement, especially by changing a person's

behaviour or the structure of something. Recent curriculum and assessment reforms in

Hong Kong purport to make a change for the better and their implementations are actions

to improve the existing conditions (Education Commission, 2000). The New Senior

Secondary Education aims to bring significant and long term benefits to Hong Kong

(EMB, 2004) while the HKDSE in the new academic structure aims to serve selection and

report student achievement explicitly. On the one hand, some local scholars posit that the

political system and internal constraints hinder education reforms in the present

environment (Morris & Scott, 2003, p.83). It is hard to forecast the success of the new

academic structure. On the other hand, after experiencing the recent policies for the two

major curriculum reforms, Target-Oriented Curriculum and School-Based Curriculum,

which had been compiled in 1990s, different stakeholders should have a thought on the

new academic structure and the corresponding new assessment framework


12/97

5

Standards-Referenced Assessment (SRA).

Some teachers have some experiences of SRA whereas others have little. For

example, some large-scale assessment programmes based on standards-referenced

frameworks have been introduced in recent times. The Territory-wide System Assessment

(TSA) was gradually introduced from 2004, first in Primary 3, the next year in Primary 3

and Primary 6, and continued to Secondary 3 in 2006. It adopts a standards-referenced

approach to assess students performance against professionally defined competences in

Chinese, English and Mathematics. Secondary teachers of Chinese, English and

Mathematics have gained some knowledge about SRA through their involvement with

TSA in which all Secondary 3 students have to participate. While other teachers do not

have this opportunity to gain SRA experience, the change in 2009 may seem more

threatening to them. Their concern about the adoption of SRA in public examinations

varies.

These differences will be compounded in the languages as a standards-referenced

system has also been implemented in the HKCE English language and Chinese language

curricula from 2005. With effect from the 2007 HKCEE, a standards-referenced

framework will be adopted and the performance of candidates will be reported with

reference to a set of five defined levels of performance (HKEAA, 2005). Descriptors

illustrating the expected performance of students at each level have been disseminated to

teachers and students and this will have enhanced understanding of SRA by language

teachers. Moreover, all teachers involved with the two language curricula will have to

attend a teacher training course to support and carry out SRA. Having more exposures to

SRA than non-language teachers at HKCEE, the language teachers may be more

competent, confident, and less concerned at the adoption of the new curricula in 2009.

Regarding SRA, secondary teachers of Chinese language and English language


13/97

6

are not only practitioners, but also users. In 2001, the EMB set up the Language

Proficiency Assessment for Teachers (LPAT), which is a five-level standards-referenced

assessment. It purports to raise the teacher professionalism and ensure the teaching

quality. It also safeguards against the public criticism of the falling language standards on

graduated students. Teachers who teach English language or Putonghua should reach

Level 3 or above in the LPAT or complete the equivalent teaching training course before

the school year in September 2006. The teachers should well understand the five defined

levels of performance and check their qualification against the standards, in order to

continue their language teaching career.

In the same way, teachers acquaintance with SRA is not limited to these

assessments and examinations. The EMB collaborating with the HKEAA had carried out

three large-scale consultation campaigns of the New Senior Secondary Education Reform

for principals, teachers and the public during 2004 - 2006. All teachers were entitled to

participate in these campaigns together with various ancillary forums and seminars

discussing curriculum change as well as the new assessment model, the

standards-referenced approach.

1.3 Teachers concerns about the innovation of SRA

The concerns or attitudes stakeholders of education reforms have about a change are an

important dimension in the change process (Hal, Wallace and Dossett, 1973). Teachers

views and concerns about the feasibility of SRA would vary. In the 2007 HKCE English

language and Chinese language examinations, teachers perceive the SRA differently. The

results of the public consultations regarding the innovation of SRA were positive and

teachers lauded the adoption of SRA. Nevertheless, some teachers publicly voice out the

difficulties they have encountered in teaching halfway through the first year of


14/97

7

implementation of the curricula and decry the new assessment framework that comport

themselves with the disavowal of the new assessment practice. Thus, they are worried

about the innovation of the SRA.

Responses to these concerns come from several directions. The innovation of SRA

implies a change in the teaching practice that teachers have to cope with not only the new

grading methodology, but also the new mode of public assessment. The introduction of

School-Based Assessment (SBA) in the two language examinations leads to a new

assessment practice in school. Teachers perceive that SBA is an add-on workload to their

current practice, and they have dissenting views on this matter although Government

officials explained it in several teachers seminars.

Moreover, teachers can perceive no significant benefits, but an additional

workload to themselves and students at the beginning of the implementation. They are

likely to have a concern and to resist the change. It seems that more time for preparation

can help lower the resistance and an appropriate timeline of implementation is needed for

teachers to well prepare the change.

Furthermore, the former HKCE English language examination had two streams,

Syllabus A and Syllabus B. Students in Syllabus A were easier to obtain a good grade than

that in Syllabus B as the standard requirement in Syllabus A is lower than that in Syllabus

B. In the new HKCE English language examination, there is only one combined syllabus.

Teachers who were used to mainly teach Syllabus A have a concern about the fairness and

appropriateness of this change to their students, the weaker cohort, who may suffer a

disadvantage from the competition with the other. The examination results and their

reporting method become the spotlight of the reform.

In addition, a high population for awarding Level 5 is set in the new grading

structure, that is more or less the same as the existing population for awarding Grades A,


15/97

8

B and C, meant to address and recognise the good performance of students. Teachers

remain sceptical about the new grading method which might evolve a portrait of grade

inflation, compared with the previous, long-life grading method. In the same way, the

new grading method might yield an unstable distribution of levels in the results over the

years and people with limited assessment literacy on SRA might think that it is not

reliable too. Thus, teachers may query the effectiveness of the new grading method and

have a concern about its reliability in the future years.

1.4 The Aim of the Study

Teachers, it can be argued, have varying stages of concern about the change of the public

examinations from a norm-referenced approach to a standards-referenced approach

because of their prior experience related to the issue. The aim of this study is to

investigate teachers perceptions of the change in the grading system of the public

examination from a norm-referenced approach to a standards-referenced approach using,

as a framework, Hall et al.s (1977) Stages of Concern (SoC) Model. This model on

stages of concern is to be discussed more fully in Chapter 2. The perceived advantages

and the potential drawbacks of a standards-referenced approach are examined in this

study.

Such an analysis is significant because it highlights the perceptions of the teachers

who are adopting and must ultimately implement SRA. It illuminates potential problems

that may occur in the process of this change and the possible directions continuing

professional development of teachers should take.

1.5 Research Questions

Given the overall aim to examine teachers views on the adoption of SRA, three research


16/97

9

questions are formulated namely:

(1) What are the stages of concern of teachers about the innovation of SRA?

(2) What factors influence the variation?

(3) What concerns appear prevalent as the hindrance to the change?

These questions are examined through a small scale survey with follow-up

interviews involving both language and non-language secondary school teachers. Details

of the methodology are given in Chapter 3.

1.6 The Structure of the Dissertation

This dissertation is divided into four further chapters. Chapter 1 introduces the study and

provides an overview of the background. Chapter 2 reviews literature on SRA and on

issues of implementation and adoption of change. Chapter 3 describes and justifies the

methodology adopted. Chapter 4 sets out the findings with regard to teachers views on

the change and the stages of concern. In conclusion, Chapter 5 discusses the findings in

the light of the literature in Chapter 2 and offers suggestions to stakeholders on the

adoption of new assessment practices.


17/97

10

CHAPTER 2

Standards Referencing and Assessment Change:

A Literature Review

2.1 Introduction

The focus of this study, as noted in Chapter 1, is on the perceptions and reactions of

teachers to the introduction of standards-referenced assessment within the public

examination system in the Hong Kong secondary school sector. It is necessary, therefore,

to examine what is understood by standards-referenced assessment.

Standards-referenced assessment is an innovation in the mainstream of the Hong

Kong school context and its adoption raises questions about factors that enhance or limit

the introduction of change within the curriculum. This leads us to the curriculum change

literature. In conclusion, the chapter summarises the key points from the review of these

two areas that are of significance to the study.

2.2 Standards Referencing

Standards referencing is gradually adopted in recent decades. Its influence to the

evolution of assessment framework is important to educational change. The nature and

emergence of standards referencing are discussed as follows.

2.2.1 Nature

All assessment activities, including public examinations, are designed for some

reasons and set in particular situations and/or specific cohorts of students. The purpose

defines the design. Public examinations are important and are major activities in many

education systems in the world. They have a long history and were first established, it is

claimed, during the period of the Sui emperors (589-618) in the school entrance and civil


18/97

11

service examinations in China (Miyazaki, 1976). At that time, public examinations were

instituted for recruiting government officials. The activity was a winnowing process and

the function of the public examination was to select elites in what was seen as a fair and

objective way so as to remove favoritism and employ the best in a meritocracy. Given that

selection was the prime purpose, the system sought to rank individuals by performance

with respect to the norm of the assessed group. Today, this would be called

norm-referenced assessment.

With a long history, norm referencing is well known for its use and application

and has been applied to the assessment of many subject areas. Norm referencing scores

represent a concise summary of students achievement relative to other students in the

cohort that can be used directly and immediately. When selection of elite is needed, this

traditional method offers a simple interpretation of scores in which the ranking can

facilitate the selection.

In context of language assessment, for example, Brown and Hudson (2002) define

a norm-referenced test as:

Any test that is primarily designed to disperse the performances of students in a

normal distribution based on their general abilities, or proficiencies, for purposes

of categorizing the students into levels or comparing students performances to the

performances of the others who formed the normative group. (p.2)

Being specialised in language teaching and applied linguistics and implicitly accepting

that the function of assessment was to rank, they elaborated that norm-referenced tests are

appropriate for assessing abstracted language ability traits representing a broader, more

differentiated content range (Millman & Greene, 1989, p.341). The broader the range the

more possible it is to discriminate high and low ranking students.

Nitko (2001, p.3) suggested a slightly different viewpoint. He described the use of


19/97

12

a norm-referencing framework as interpreting students assessment performance by

comparing it to the performance of a well-defined group of other students who had also

taken the same assessment. The focus of his notion was to identify the relative strengths

and weaknesses of students in their cohort, rather than individuals achievements. Or it is

more like making a relative decision (Brown & Hudson, 2002, p.2) in the assessment

result.

Whenever assessment takes place, a referencing framework is needed (Nitko,

2001) to interpret students performance in assessments in general and in public

examinations in particular and, as noted, the form of referencing must match the purpose

of assessment. Brown and Hudson (2002) suggest four different types of focus for

classification: norm-referencing, criterion-referencing, domain-referencing and

objectives-referencing. Domain-referencing and objectives-referencing are, however,

variations of criterion-referencing and Brown and Hudson note that much confusion has

also been generated by the different ways that criterion-referenced has been used, as well

as by the alternative terms that have evolved for different types of tests closely related to

criterion-referenced tests (p.3). In the same way, Nitko (1980) distinguishes the many

varieties of criterion-referencing assessment by revealing various conceptualisations it

can take.

Sadler (2005) has conceded that the meaning of criterion is broad in scope.

However, an assessment criterion is a statement of the basis against which the assessment

is made. A standard is a form of criterion. Criterion referencing as originally conceived

was based within a measurement paradigm and stressed that criteria should be specified

only where appropriately reliable measurement was possible. This produced narrow

criteria. In a learning environment, broader criteria were adopted but this caused

confusion as to the meaning of the term criterion. Hence the notion of standards was


20/97

13

developed as a more broadly based concept. This is discussed more fully in the next

section. At the same time as considering the general definition of standard, Sadler (1987)

discerned a major shift in orientation from criterion-referencing per se to

standards-referencing so as to realise the aspirations of criterion-referencing (Sadler,

2005).

Norm-referencing, criterion-referencing and standards-referencing are now widely

used in many public examination systems in the world, but in recent years there has been

a shift away from the first. Nitko (2001) has commented that norm-referencing is not

enough to fully interpret scores, noting that:

Norm-referencing provides information helpful for your relative interpretations

of scores, but frequently these are not enough. Scores that reflect relative

achievement such as rank order, for example, may be helpful in picking the best

readers, or in sectioning a class into better, good, and poor readers. However, to

plan appropriate instruction, eventually you need to know the kinds of reading

performances each student can do and the particular types of difficulties each

student is experiencing. (p.3)

He signified that interpretation of scores requires more information for effective selection

and ranking of students and defining students level of achievement. He, however, was

not the first to articulate such defects in norm referencing, but he extended Robert

Glasers thinking documented in his seminal paper in 1963. Glaser (1963) introduced the

phrase criterion-referenced measurement for evaluating students performance. Over

the years, criterion-referenced assessment and criterion-referencing have come to be

used to represent his ideas and these phrases have now become integral to the

nomenclature of assessment.

Glaser (1963) pointed out that norm-referenced assessment merely revealed the


21/97

14

relative performance, but he sought to emphasise what information could be obtained

from a norm-referenced score commenting that:

Along such a continuum of attainment, a students score on a criterion-referenced

measure provides explicit information as to what the individual can or cannot do.

Criterion-referenced measures indicate the content of the behavioral repertory, and

the correspondence between what an individual does and the underlying

continuum of achievement. Measures which assess student achievement in terms

of a criterion standard thus provide information as to the degree of competence

attained by a particular student which is independent of reference to the

performance of others. (pp. 519-520)

He argued that criterion-referencing was the only referencing system for all assessments

and viewed norm-referencing as a variation of criterion-referencing in which criteria were

set in relation to the mean score.

Popham and Husek (1969) also enumerated the differences between

norm-referencing and criterion-referencing. They define a criterion-referenced test as that

used to ascertain an individuals status with respect to some criterion, i.e. performance

standard (p.2). Thus, a criterion-referenced framework is not simply a change of scoring

practice or a different interpretation of scores. It is qualitatively a different perspective.

Glaser and Nitko (1971) agreed with Popham and Husek (1969) that such a change could

not be made by simple inspection of a particular instrument but was linked to teaching.

They emphasised the domain of instructionally relevant tasks in which assessments take

place. In other words, these scholars inclined to the view that the domain of knowledge

regarding instructional objectives should also be considered in any particular assessment

framework.

At the same time, Garvin (1971) introduced the idea of subject-matter areas to


22/97

15

the criterion-referenced framework that subject matters might affect the application of this

framework because performance criteria could not be set in some conditions. Popham

(1973) added his idea on writing instructional objectives for teachers that

There is, however, another dimension to objective writing, a dimension that

further aids the teacher in planning and evaluating his instruction. It involves

establishing performance standards, that is specifying prior to instruction the

minimal levels of pupil achievement. (p. 3)

Hambleton and Sireci (1997, p. 383) summarised the situation writing that students test

scores were often interpreted with respect to the performance of other students, rather

than with respect to their attainment of desired educational objectives in the basic skills

testing movement from the 1970s and 1980s. Teachers and students often desired

diagnostic information for improving performance in such a way as to link to mastery of

important outcomes of instruction (often called objectives). For this reason, criterion

referencing increasingly became important.

2.2.2 The emergence of standards-referencing

As noted, traditionally, assessment practices sought to comply with a simple,

direct goal of selection, a goal which stakeholders including teachers, parents and

students accepted with alacrity. However, in recent years, criticisms (Glaser, 1963;

Popham, 1978; Hambleton, 1996; Nitko, 2001) have emerged. Traditional assessment

practices have been seen a narrow and narrowing in terms of learning as well as providing

only limited information on what has been achieved. The term educational testing has

consequently been replaced by educational assessment (Hambleton, 1996, p.899) and

the term assessment has been broadly interpreted as to include any judgment (or

appraisal, or evaluation) of a students work or performance made by a teacher or other

competent person, whether for purposes of improvement or certification. (Sadler, 1987,


23/97

16

p.191) The focus has shifted to the instructional objectives within teaching and learning.

There has been what has been called a paradigm shift from the dominance of

measurement for selection (Gipps, 1992) to assessment for learning and, where

assessment is of learning as in public examinations, to ensure the assessment supports

learning as far as possible.

A criterion-referenced framework, as indicated earlier, is argued to be able to

fulfill such aims for assessment and overcome the aforementioned issue. Scholars and

educators have been keen to try it out and many developed countries have changed and

adopted a criterion-referenced framework in their public examinations. For example, in

the National Curriculum for England and Wales formulated in 1988, assessment criteria

were set out for the illustration of students attainment targets. It has also been adopted in

Australia, New Zealand, South Africa and in North America.

Sainsbury and Sizmur (1998) studied the meaning of criterion-referencing in the

UK National Curriculum approach and pointed out that:

Criterion-referenced assessments, in an educational context, aim to give

information about valued educational outcomes. They aim to tell us how well the

pupils have learned what they have been taught. They are typically about

cognitive outcomes, with understanding, knowledge and skill as central elements.

This understanding, knowledge and skill is often described in fairly abstract and

general terms. (p.3)

Notably, they were of the opinion that such an educational construct was abstract, general

and complex. In general, drawing criterion-referenced inferences from test scores means

claiming links from one extreme to another. (Sainsbury & Sizmur, 1998, p.3) This

educational construct, the criterion-referenced framework, failed in practice as Sainsbury

and Sizmur (1998) felt that there were difficulties of teacher workload, expressed in


24/97

17

various ways over the years and culminating in a full-scale boycott in 1993. The

difficulties were identified as related to the complexity of the original, statement of

attainment based, assessment system (Dearing, 1993).

Earlier, in 1987, Royce Sadler highlighted the difficulty of adopting

criterion-referencing in public assessment and raised some concerns about its general

applicability, though many methods had been proposed for dealing with scores and grades

in large scale examinations (Berk, 1986). He clarified the terms criterion and

standard. Criterion-referenced assessment is deliberately constructed to yield

measurements that are directly interpretable in terms of what was specified performance

standards. (Glaser & Nitko, 1971, p.653) Glass (1978), however, interpreted the term

criterion in a different way and shifted its meaning to an arbitrary, minimal level of

competence.

Sadler (1987, p.195) suggested that:

Originally, a criterion referred to a characteristic or dimension of performance,

but since about the time of Glaser (1963), a criterion has been used to mean the

particular score that is taken to designate competence or mastery. The criterion in

criterion-referenced testing is invariably a numerical cut-off. According to the

above definitions, therefore, it would be better called a standard.

He (1987) proposed an influential assessment framework which he termed

standards-referencing for the certification of students complied with both quality control

and the due recognition of achievements. He also contended that practical considerations

should be in the context when dealing with standards (pp.196-197).

According to Sadler (1987), in order to forge strong links between grades and

actual achievement, observation of performance is of paramount important in

standards-referenced assessment. The original National Curriculum for England and


25/97

18

Wales, which adopted the criterion-referencing, was then improved by introducing a

better judgment mechanism, after the original scheme led to a boycott in schools.

Sainsbury and Sizmur (1998) wrote that the level descriptions which were proposed in the

improvement in 1995 replaced the simple statements of attainment; the mode of use of the

level description was to form a best fit judgment. Although the current National

Curriculum for England and Wales is claimed to be criterion-referencing framework, the

change in practice after the instance of boycott reveals the shadowy side of a

standards-referencing framework defined by Sadler (1987). Others, however, have

described it as no more than a weak version of criterion-referenced assessment (Gipps,

1994, p.95).

In addition, Gipps (1994, p.85) alleged that:

A major problem for criterion-referenced assessment is aggregation, i.e. the

collapsing of the detailed performance profile for each individual into a single

report figure, or grade. Aggregating detailed assessment information into a crude

single grade compromises the information offered by the assessment. A final

summative aggregated grade does not help the pupil or employer: it obscures more

than it clarifies.

She raised queries about the validity and reliability of criterion-referenced assessment.

She felt that strict criterion-referenced assessment was an unmanageable and undesirable

educational assessment framework (p.96). She also commented that weak versions of

criterion-referenced assessment or standards-referenced assessment defined by Sadler

(1987), that were being pursued in the USA and in Australia, had huge problems in terms

of comparability across assessments.

Standards-referenced assessment thus has many potential benefits over a norm

referenced system but as a system for public examinations is noted to have conceptual


26/97

19

and practical difficulties. It is perhaps not surprising that adoption creates issues of

curriculum change which are now discussed in the next section.

2.3 Curriculum Change

In this section, we review literature on curriculum change and on understandings of the

challenges involved in the process.

2.3.1 Conceptions of curriculum change innovation or adaptation?

Assessment change is a part of curriculum change. Change can be referred to an

update of existing elements or replacement by a new entry. Fullan (2001) avowed that

curriculum change should embrace everything new to stakeholders including knowledge,

teaching strategies, and beliefs. Stenhouse (1975) concurred arguing that knowledge and

teaching strategies were the key elements in curriculum change. He did not mention

assessment per se, but it may be viewed as a part of teaching and learning. He (p.169) was

concerned about the views and identity of teachers during change, and suggested updates

of these elements that were more easily accommodated than introducing new but

disturbing elements to teachers.

In the context of this study, this highlights a potential problem as standards

referencing marks a radical departure in practice rather than an evolution. Cuban (1979)

concurred and suggested that curriculum change should try to improve the existing

elements and perform incremental change rather than fundamental change (Cuban, 1997).

The implementation is also a noteworthy consideration to curriculum change.

On the one hand, curriculum change is needed when the current practice has some

recognisable weaknesses and does not fit the present environment. Curriculum change

provides a new setting or platform for innovation to grow. On the other hand, curriculum

change is introduced to bring about social change and society perceives the curriculum


27/97

20

change as being necessary. However, Morris and Scott (2003) have discussed some

instances of curriculum change in Hong Kong and indicated that policy makers have

proposed curriculum changes for economic, social, and political reasons, and that

proposals often become rhetorical slogans rather than agendas for change when the

primary reason is not related to pedagogical perspective.

2.3.2 Curriculum change strategies

Assessment change in the public examination context is often led by government

or quasi-government organisations. Curriculum change as an educational process has

been studied for well over 40 years. Scholars like Schn, Havelock, Hargreaves, Bennis,

Benne and Chin have developed influential models on change strategies that may be

relevant to assessment change issues being considered in this study. Their models have

one thing in common. They all appreciate the top-down strategy in different parlance.

Schn (1973), Havelock (1973), Hargreaves (1994), and Bennis, Benne and Chin (1985)

respectively use centre periphery model, Research, Development and Diffusion model,

cultural interruption model and empirical-rational model to describe a similar idea that

leadership is centralised and change is disseminated to the periphery or inferiors by

reasoning.

Nevertheless, those scholars emphasise different approaches. Schns ideas (1973)

on curriculum change focus on diffusion of innovation. He comments that

The loss of the stable state means that our society and all of its institutions are in

continuous processes of transformation. We cannot expect new stable states that

will endure for our own lifetimes. (p. 28)

He asserts that a continuous process of transformation will be taken place and the stable

state or centre will then be lost. He therefore develops proliferation of centres and

shifting centres models to echo the phenomenon of the transformation.


28/97

21

Havelock (1969) was originally concerned about the dissemination of knowledge

emphasising the characteristics of the change agent and explaining how change occurs.

He incorporated the idea of diffusion of innovation into the notion of the dissemination of

knowledge, and develops the concept of linkage (Havelock, 1973), social interaction

and problem solving. In the social interaction model, a five-stage cycle of awareness,

interest, evaluation, trial and adoption is codified to identify the strategy of change. The

concept of linkage focuses on the users of innovation as problem solvers. He argued that

problem solvers, as opposed to observers, will utterly and proactively involve themselves

in developing the innovation concerned.

Hargreaves (1994) focuses the strategy on changing teachers in a postmodern age.

He postulates that teachers have to adapt to a new school reality when a new culture

arises. Generally, it is named as cultural interruption or classified as a notion of

interrupting the existing culture. He also develops structural reinforcement model to

describe a dictatorial strategy. Sometimes a structural redefinition is needed to tackle the

alienation and marginalisation of teachers (Hargreaves, 1994, p.242). Bennis et al. (1985),

however, developed an empirical-rational strategy to illustrate the top-down approach

where reasons and cohorts interest are encompassed.

2.3.3 Teachers as change agents

Teachers, in the past, were expected to replicate predefined instructional tasks in

school. However, in the recent decades, education reforms are evolved and there is a shift

in teachers role. Many curriculum scholars (Cuban, 1988; Fullan, 2001; Hall & Hord,

1987; Oliver, 1977; Rudduck, 1991; Stenhouse, 1979) concern teachers as the major

players in education reform. They concur that teacher is the key role in curriculum change

despite the aforementioned top-down curriculum change strategies, and describe teachers

as change agents in a sense that facilitate the process of change.


29/97

22

Cuban (1988) emphasised the vision of change, positing that teachers should share

the vision so as to lead to a success in curriculum change. Stenhouse (1979) focused on

the way from the teachers point of view and stated the axiom: No curriculum

development without teacher development. He confirmed the importance of teaching

involvement and the development should make teachers feel a sense of ownership of the

curriculum. Oliver (1977) also agreed that successful curriculum change should involve

teacher participation. Hall and Hord (1987) stressed that teachers not only have the

ownership and participate in the change, but also are able to moderate the change or be

facilitators of the change. Rudduck (1991), however, argued that ownership is fragile and

is difficult to be measured, though he articulated the importance of teacher beliefs on

changes in practice.

In brief, the involvement of teachers is highly regarded as an essential trait in

education reform. Their concerns about and perceptions of and their actions and reactions

to a curriculum change suggested with standards-referenced assessment are thus crucial

and are discussed in the next section.

2.4 Teachers perception and Concerns Based Adoption Model

Understanding teachers concerns and their level of use is vital in any issue of

implementation as is considered here in the case of assessment framework. In regard to

standards referencing, there seems relatively little research related to teachers perception

of SRA. However, in 2003, Ayres, Beechey and McCormick investigated teachers

perceptions of the impact of the New NSW Higher School Certificate in New South

Wales, Australia including the change of assessment framework to using standards

referencing. They designed a three stage longitudinal study within a theoretical

framework derived from established theories of motivation and occupational stress. They


30/97

23

found that teachers perception helped them analyse the issues of curriculum change

including speed of implementation, workload and teaching strategies.

Over the decades, there have been various major, prominent approaches to

researching teachers concerns and reactions namely Action Research, Concerns Based

Adoption Model (CBAM), Curriculum Alignment, and Comprehensive School Reform

Programs (Marsh & Willis, 2003, p.249). The CBAM is a widely applied theory and

methodology for studying the process of implementing education change by teachers

(Anderson, 1997) and is a focus in this study.

Table 2.1 Typical expressions of concern about an innovation

Stages of Concern

6 Refocusing

5 CollaborationImpact

4 Consequence

Task 3 Management

2 PersonalSelf

1 Informational

Unconcerned 0 Awareness

The concept of CBAM was germinated in the work of Frances Fuller in 1969.

Meanwhile, others in response to the innovation focus approach to educational change

evolved (George, Hall & Stiegelbauer, 2006). CBAM is composed of Stages of Concern,

Levels of Use, and Innovation Configuration (Hall & Hord, 1987). In particular, seven

Stages of Concern about an innovation categorised in three groups are identified and a

corresponding 35-item questionnaire, namely Stages of Concern Questionnaire (SoCQ),

is developed by Hall, George, and Rutherford (1977). SoCQ can be used as a diagnostic


31/97

24

tool to assess teachers perception in relation to the adoption of educational change. The

seven Stages of Concern about an innovation and their three corresponding groups are

shown in Table 2.1.

In some reviews of CBAM (Anderson, 1997; Cheung, Hattie, & Ng, 2001),

scholars agreed that SoCQ, as one diagnostic dimension of CBAM, is a possible tool for

measuring teacher concerns about a change in curriculum or instruction. However, they

reported some critical notes and even suggested modifications to SoCQ to cater for the

need in their research. Anderson (1997, p.334) noted that CBAM theory idealises the

Stages of Concern as a developmental progression in which teachers implementing a

change have concerns of varying intensity across all seven stages at different points in the

change process. He concluded that the resolution of early stage concerns does not

necessarily lead to the arousal of later stage concerns about the impact of those practices

on students.

Moreover, Cheung et al.(2001) found an imperfection in SoCQ from a statistical

point of view. They modified the original SoCQ and constructed a 24-item questionnaire

to measure teacher concerns clustered in five sequential stages (Cheung, 2001). George et

al.(2006) also comment the weakness of SoCQ noting that

Development of higher-level concerns cannot simply be engineered by an outside

agent. Holding concerns and changing concerns is a dynamic of the individual.

Providing affective experiences and cognitive resources in a timely manner

certainly can supply the grist for the emergence and resolution of concerns,

thereby facilitating the development of higher-level concerns. There is no

guarantee, however, that the emergence of higher-stage concerns will follow the

reduction of lower-stage concerns.

To sum up, there are criticisms to the use of CBAM yet many studies have


32/97

25

demonstrated how effective CBAM can be to recognise the inevitable presence of

concerns within individuals and to extend a helping hand to assist in copying with and

resolving those concerns (George et al., 2006).

2.5 Conclusion

This chapter enumerates the reasons and the potential benefits, found by academics, of

the adoption of standards-referenced assessment in many education systems including

Hong Kongs. To implement or to execute the innovation in a curriculum change, there

are a number of possible top-down curriculum change strategies that could be applied,

and teachers play as change agents in the process, as magnified in different perspectives.

Concerns of teachers can then be measured through the CBAM to codify the teachers

perception in a system way that corresponding, appropriate policies can be made or

revamped to avoid any hindrances to the change, which will be discussed in the chapters

followed.


33/97

26

CHAPTER 3: METHODOLOGY

3.1 Introduction

This chapter discusses the methodology used in the evaluation of teachers attitudes to the

introduction of new assessment approaches. As noted in Chapter 1, concern is an

important dimension when working with individuals involved in a change process

(George et al., 2006, p. 1). In the change process involved with a shift in the public

examinations from a norm-referenced approach to a standards-referenced approach,

teachers are the practitioners who play an important role to the success of the reform.

Teachers with or without prior experience may have various kinds and degrees of concern

about the adoption and implementation of standards-referenced assessment. The aim of

this research, as noted in Chapter 1, was to investigate the impact of prior experience on

the concerns of secondary school teachers about the innovation of SRA. Are there

different stages of concern regarding SRA between those to whom, this is new as opposed

to those who have been working with it for a few years? The three key research questions

are:

(1) What are the stages of concern of teachers about the innovation of SRA?

(2) What factors influence variation in concern?

(3) What concerns appear prevalent in hindering change?

To carry out a logical and reliable study, the Concerns-Based Adoption Model

(CBAM) was adopted as a framework for collecting and analyzing teacher concerns

through questionnaire. This chapter describes and explains how the CBAM was

operationalised as the framework in this case. In particular, it discusses issues in the use

of interview as a data collection tool and how the data collected was analysed.


34/97

27

3.2 CBAM Approach

The CBAM is a diagnostic model for analyzing an education change in schools which

primarily focuses on the concerns of teachers (Hall, Wallace, and Dossett, 1973). It

comprises three elements: Stages of Concern (SoC), Levels of Use (LoU), and Innovation

Configuration (IC). SoC is used to evaluate the concerns of teachers when they are

going to implement an innovation, while LoU is used to find out what teachers have done

in the implementation. IC is used to help understand the difference in ways that teachers

modify and implement innovations. These three components, as three different

dimensions, reflect a diagnosis of an education change and a means to research change.

Table 3.1 Typical expressions of concern about an innovation

Stages of Concern Expressions of Concern

6 I have some ideas about something that would work even better.

5I would like to coordinate my effort with others, to maximise

the innovations effect.Impact

4 How is my use affecting my students?

Task 3 I seem to be spending all my time getting materials ready.

2 How will using it affect me?Self

1 I would like to know more about it.

Unconcerned 0 I am not concerned about it.

This research focused on Stages of Concern (SoC), one of the diagnostic

dimensions of CBAM, to understand teacher concerns. Since the 1970s, SoC has been

used in many studies. The concepts and knowledge that it reflects are well known in the

literature. It acts as a common language for researchers, policymakers, and users to talk

about the implementation of innovations (Snyder, Bolin, and Zumwalt, 1992). Particularly,

having a primarily descriptive and predictive approach, SoC is suitable for research on the

innovations about understanding curriculum implementation (Anderson, 1997). Typical


35/97

28

expressions of concern about an innovation, suggested by George et al. (2006, p. 4), are

shown in Table 3.1.

The CBAM assumes that teachers undergo similar concerns in a change process in

such a way that teachers, starting at an unconcerned stage, experience a sequence of

changes in concerns about self, task, and impact. These teachers move their concerns

from personal issues to teaching, and then students themselves. This unidirectional

progression of teacher concerns about a change may be possible, but not necessary at all

time (Leithwood and Montgomery, 1982). Some researchers (e.g. Bailey and Palsha, 1992;

Cheung et al., 2001), on the other hand, point out the universal, fixed, discrete stages of

concern. They allege that SoC is not always appropriate to illustrate the teacher

concerns without the consideration of the authenticity in different scenarios. It leads to a

question of potential validity. For this reason, these researchers have refined and revised

SoC for its use in their specific situations.

No matter SoC is modified or not, it has been widely used in many seminal studies

and this demonstrates an effective methodology to collect and describe teachers concerns

clearly. The model, however, may suggest directions that can be pursued to find

interpretation. But it does not directly lead to the explanation to the findings. To put it

simply, the model cannot directly explain why teachers have a certain stage of concern,

and yield the reasons behind this. Hence, other research tools like interview adopted in

this study are required for this purpose.

3.3 Research Design

A naturalistic, qualitative case study design was adopted in this research. Such a design

has several hallmarks. It can provide rich and vivid descriptions of the events relevant

to the case and blends the descriptions of events with analysis of them (Hitchcock and


36/97

29

Hughes, 1995), as presented in this study. The conceptual framework was developed

and shown in Figure 3.1. It illustrated that data are collected from teachers with or

without prior experience of SRA and classified into teachers perception of change and

perceived hindrances regarding the assessment change or the innovation of SRA. It also

manifested that reasons were deduced and analysed for further understanding with the

change and developing implication for practice.

Figure 3.1 Conceptual framework

There were two phases in data collection. In the first phrase, the CBAM Stages

of Concern Questionnaire (SoCQ) were used as a quantitative method to assess teacher

concerns. In the second phrase, with the preliminary results of the teacher concerns

about the innovation of SRA, five interviews were conducted to seek interpretation of

teacher concerns, and to strengthen and delineate a wider, in-depth scope of research

findings.

Perceived

hindrances

Perception

of change

Teachers

With prior experience

of SRA

Without prior

experience of SRA

Implications

for practice

Reasons

Reasons


37/97

30

In the first phase, the SoCQ (described in the next section) was distributed to

teachers. The questionnaires were firstly classified into two cohorts: language and

non-language teachers and then by teaching experience. Data analysis on the SoCQ was

based on the SoCQ Quick Scoring Device of George et al. (2006, p. 86) and used to

calculate percentile scores for plotting Stages of Concern profiles. From this, variation

of stages of concern between different cohorts could be assessed, where the researcher

could gain a great deal of insight, not only into the types of concern that were most

intense and least intense, but also into the affective stance that the respondents were

taking towards the standards-referenced assessment (George et al., 2006, p.37).

After the completion of this survey, five interviews were conducted in the second

phase. The five interviewees were secondary school teachers, three of whom were

language teachers. Before starting the interviews, they had to complete a SoCQ to

delineate their individual Stages of Concern profile.

The researcher arranged and met five teachers for the data collection of the

questionnaires and interviews in two months. He mainly employed the interview guide

approach suggested by Patton (1980, p. 206) at first. Nine semi-structured questions, as

shown below, were designed for the teacher interviews.

1. What are your greatest concerns about the standards-referencedassessment?

2. What type of support do you think you need from EMB/HKEAA in orderto be successful?

3. What type of support do you think you need from school/mentor in orderto be successful?

4. What type of support did you receive that was most helpful in assessmentreform?

5. How does your experience compare to your expectations prior to teaching?6. Do you see the standards referencing adopted in the examination as a loss

or an improvement? Why?


38/97

31

7. What keep you away from learning more about standards-referencedassessment?

8. What would help you most to continue to learn about the use ofstandards-referenced assessment?

9. Is there anything else you would like to say about the standards-referencedassessment?

The questions covered the related topics and issues which were constructed in advance,

and the researcher decided the sequence and working of the questions in the course of the

interviews, so as to collect interviewees responses and opinions accordingly.

When the data collection of the SoCQ was completed, Stages of Concern profiles

for different cohorts and individual interviewees were studied and collated with the

analysis of the qualitative data collected from the interviews. Through the quantitative

and qualitative analysis, it was hoped that the findings could apply theories to fit the

situation, following advice suggested by Lincoln and Guba (1985) to maintain the

credibility of the research.

3.4 The Operationalisation of Stages of Concern Questionnaire

This section describes how the data of the questionnaires was collected in the study. It

includes the design of the questionnaire, the selection of participants, the administration

procedures, and the presentation of results.

3.4.1 The Stages of Concern Questionnaire

The researcher adopted the Stages of Concern Questionnaire (SoCQ) developed

by Hall et al. (1977) but with the changes for this particular study as set out below. The

questionnaire (see Appendix A) was organised in two parts: the introductory page and two

pages of items. The introductory page stated the purpose of the questionnaire and

explained how to complete the items. A small number of questions were added to collect

the personal information of respondents, these being necessary to uncover the teaching


39/97

32

experience of respondents which arguably was an influential factor. This facilitated data

analysis and the impact of prior experience on the standards-referenced assessment.

Respondents noted their teaching experience in English/Chinese language teaching and/or

other subjects. Language teachers were defined as teachers who taught English language

or Chinese language class in Form Four or above, because this was where

standards-referenced assessment had been introduced. Five option boxes with different

ranges of years of teaching experience were provided: nil, 0-2 years, 2-4 years, 4-6

years, and 6 years above. Six years of teaching experience was decided as a

maximum as teachers with six years of teaching experience were regarded as experienced

teachers because the secondary teachers who taught Form Four class or above were in the

Graduate Master/Mistress level, and in practice, they generally have at least five years of

teaching experience and have undergone professional training to get promotion to the

Senior Graduate Master/Mistress level. This meant they gained sufficient teaching

experience as an experienced teacher to take up more important school duties such as

panel chairperson and discipline master/mistress. In addition, optional name and email

were requested in the questionnaire.

On the next two pages, 35 items were included. The order and the wording of

these items were preserved from the original SoCQ (Hall et al., 1977) to avoid the risks

pertaining to reliability and validity (George et al., 2006, p.25). However, for better

understanding as suggested by George et al. (2006, p.25), the original word in the items,

such as innovation, was replaced by the more specific reference to standards-referenced

assessment, which was a familiar term to and a closer bonding with the respondents.

For example, Question 1 in the SoCQ was

I am concerned about students attitudes toward the standards-referenced

assessment.


40/97

33

with the replacement of innovation by standards-referenced assessment.

Respondents circled each item on a 0-7 Likert scale according to how true the

item seemed to them at the moment they completed the questionnaires. The 0 as the

smallest value on the scale represented that the item was completely irrelevant to the

respondents, while the 7, as the extreme value on the scale, represented that the item

was strongly a truism to the respondent.

In the questionnaire, there were no open-ended questions. The reasons were

twofold. First, there were problems of data handling (Cohen, Manion and Morrison, 2000,

p.255). On the one hand, while data collected from open-ended questions are word-based,

respondents might write down their expressions in different ways so that it would be

difficult for the researcher to make comparisons between respondents. On the other hand,

it was questionable how to aggregate such qualitative data into somehow discrete,

confined quantitative data as the results generated from the existing 35 items. Second,

with a better interpretation of the questions to which respondents answered unclearly or

incompletely, the researcher should carry out follow-up activities such as sending

follow-up letters to maximise the responses from hundreds of respondents (Cohen et al.,

2000, p.263). However, this might not be possible to accomplish in a timely fashion.

3.4.2 The selection of respondents

This research focused on the assessment change in the new language curricula in

Form Four and Form Five levels. The target respondents were teachers who had

experience on teaching senior classes in school. Questionnaires were distributed to

twenty-two teachers in different secondary schools who completed them and attempted to

redistribute the questionnaires equally to languages and non-language teachers in their

schools. The initial twenty-two teachers were chosen in the study because they were

working closely with the researcher and the researcher could easily approach them. This


41/97

34

represented a convenience sample, which is a possible sampling strategy for a case study

(Cohen et al., 2000, p.103) like this research. At the same time, however, the schools

were of different types including government school, aided school, caput school, and

private school. Beside themselves, they asked close colleagues to fill in the questionnaires.

It was very important that the respondents were encouraged, not coerced into completing

a questionnaire if reliable and valid responses were to be obtained. With the liaison

between the researcher and the twenty-two teachers, respondents were given appropriate

care and incentives to accomplish the questionnaires so as to maximise the response rate

following advice in Cohen et al.(2000, p.263). In total, about 450 questionnaires were

distributed through the twenty-two teachers in the entire exercise.

3.4.3 The administration procedures

The twenty-two participating teachers were provided with a detailed briefing as to

the purpose of the exercise and the researcher asked for their help on a voluntary base.

The researcher tried his best to explain the helpers the nature of the research so that they

could distribute the questionnaires to the right persons. He also explained the helpers their

involvement in this exercise in order to avoid any misunderstanding regarding ethical

issues such as the sensitivity of the questionnaire and data privacy (Cohen et al., 2000, pp.

245-246). Moreover, he closely communicated with the teacher helpers by email and

phone to answer their queries on matters arising for minimal misinterpretation.

After respondents completed the questionnaires, some teacher helpers put the

questionnaires into a return envelope provided by the researcher and sent it out by postal

mail. The others directly passed the questionnaires to the researcher by hand. Of 450

questionnaire copies, 259 questionnaires were completed and returned in June 2006.

The return rate was satisfactory as more than 50 per cent response (58%) to the

questionnaire was received (Cohen et al., 2000, p. 246).


42/97

35

The SoCQ was well developed for decades by scholars. Its validity and

reliability relied on such a way that the 35 items were inter-correlated, meaning to the

equal essence of each item. In this research, missing data in SoCQ was defined as no

response to any items including the first one on the introductory page or multiple

responses to any one of the 35 items. Before the data entry process, the researcher

decided to void any questionnaire with missing data. It was because out of 35 items in a

SoCQ, no matter how many correct items were there, this kind of questionnaire had

outlived its usefulness and would affect the validity and reliability of the data analysis

(Cohen et al., 2000, pp. 245-246). When the data in the questionnaires were being entered

into a database program, it was found that some items in a small portion of the collected

questionnaires were completed by either double circles or no circles. After the data

entry process, six questionnaires were found to be voided. Hence, 253 valid

questionnaires were compiled in this study and 135 of them were filled in by language

teachers, who taught Chinese language or English language in Form Four class or above

in their schools.

The database program based on a database software package for data processing

was developed by the researcher. It was used to manage the item data of the

questionnaires and to administer the data analysis. Most of the data were values of the

35 items in the SoCQ ranged from 0 to 7. The program was designed with check sum,

type check, and range check to minimise data input errors. When data were input into

the program, the program could automatically and instantly detect typing errors and it did

not accept entries with missing data or incorrect data such as letters, symbols, and

numbers 8 and 9, by prompting the user to enter the data again.

Besides, a spreadsheet file consisted of preset formulae and macros which was

developed by George et al.(2006) was used for data analysis. Based on the SoCQ


43/97

36

Quick Scoring Device (George et al.,2006, p. 86), the worksheets in the file calculated

raw score totals and percentile scores. Subsequently, those percentile scores were

compiled for plotting Stages of Concern profiles. The Stages of Concern profiles for

different cohorts were plotted accordingly and were represented by the mean percentile

scores of the individuals in a group (George et al., 2006, p.34). The cohorts included

language teachers, non-language teachers, experienced language teachers, inexperienced

language teachers, experienced non-language teachers, inexperienced non-language

teachers, and the whole group. The profiles were used as a tool to generate an overview

for diagnosing the factors involved in the variation of teacher concerns.

3.4.4 The presentation of results

The data collected from the SoCQ was interpreted and translated into meaningful

Stages of Concern profiles by the spreadsheet file as mentioned in the previous section.

Figure 3.2 An example of Stages of Concern profile

0 1 2 3 4 5 6

0

10

20

30

40

50

60

70

80

90

100

Awaren

ess

Inform

ation

al

Pers

onal

Man

agem

ent

Cons

eque

nce

Collabo

ration

Refocu

sing

Stages of Concern

RelativeIntensity

Figure 3.2 is an example of Stages of Concern profile showing the relative intensity of the

subject in each of the stages of concern. A Stages of Concern profile was a


44/97

37

two-dimensional graph with Relative Intensity ranging from 0 to 100 against the Stages

of Concern: Awareness (0), Informational (1), Personal (2), Management (3),

Consequence (4), Collaboration (5), and Refocusing (6).

3.5 Follow-up Interviews

This section describes how the data of the interviews was collected in the study. It

composes of the purposes of the interviews, the formats of the interview questions, the

selection of interviewees, the administration procedures, and the presentation of results.

3.5.1 Purposes of interview

Interview is a powerful tool in academic research. It is not simply a unidirectional

way of collecting data as questionnaire. It enables an interaction between interviewers

and interviewees to discuss their interpretations of the subjects they concern. As Cohen et

al. (2000, p.268) have indicated that interview not only serves as a means of gathering

information having direct bearing on the research objectives. Interview can be used to

suggest hypotheses and in conjunction with other methods in a research undertaking.

In the exercise of interviewing teachers in this research, the interview guide

approach was used. As Patton has said (1980, p.206), the interview guide approach

represents a systematic way to collect comprehensive data for each interviewee in which

logical gaps in data can be anticipated and closed. This exercise purported to sample

teachers views so that the researcher could triangulate it with the result of questionnaire,

and seek to develop hypotheses for answering the research questions.

3.5.2 Format of interview questions

The research objectives were translated into semi-structured, open-ended

questions for interviewing the five teachers. Most of the questions were asking for the

interviewees personal experiences and feelings. For example, Question 7 is,


45/97

38

What keep you away from learning more about standards-referenced

assessment?

Interviewees could give the answers of different aspects. Ostensibly the reasons that kept

interviewees away from learning more about standards-referenced assessment might be

something related to their personal professional development or obstacles they faced in

school in terms of students or other issues. This open-ended question allowed

interviewees a lot of flexibility to express their views. This kind of question can help the

research learn about interviewees understanding in greater details and truly assess what

they really believe. It may result in unexpected or unanticipated answers which may

suggest unthought-of hypotheses (Cohen et al., 2000, p.275).

The sequence of the interview questions was arranged in a way that easier and less

threatening, non-controversial questions are addressed earlier in the interview for putting

respondents at their ease (Patton, 1980, pp.210-211). That was why the first four

questions were what-type questions, the easiest one as warm-up questions, while

how-type and why-type questions were put forth later.

The first interview was planned to have dual functions. Besides collecting data of

the interviewee, it was treated as a trial run to benefit the rest of the interviews. This trial

run purported to fine-tune the original set of questions and the way that the interviewer

asked questions and managed the interviews, so that the interview questions could

adequately reflect what the research objectives were expected to be done.

3.5.3 Characteristics of interviewees

Similar to the said operationalisation of Stages of Concern Questionnaire, the

focus of the interviews was on the assessment change in the new language curricula in

Form Four and Form Five levels. The criteria to select target respondents were the same.

The interviewees should have experiences on teaching senior classes in school.


46/97

39

As a result, it was planned to select and invite five acquaintances of the researcher

to participate in the interview part of this study to collect different views on the

innovation of standards-referenced assessment. They were invited not because it was a

convenient or casual selection, but because they basically fulfilled the preset criteria.

However the researcher had confidence to conduct the interviews with them in a way that

an appropriate atmosphere could be established for them being situated for good

interaction and communication (Cohen et al., 2000, p279). The five interviewees had also

participated in helping the distribution and collection of the SoCQ. So, they had some

basic knowledge about the research as well as the interviews beforehand. It facilitated the

conduct of the interviews indeed.

Table 3.1 Teaching backgrounds of interviewees

Interview Name Subjects taught Teaching experience

Senior form: Geography1 Rachel

Junior form: English language12 years

Senior form: English language2 Karen

Junior form: English language8 years

Senior form: English language3 Ida

Junior form: Chinese language5 years

Senior form: Geography, computer4 Lee

Junior form: Geography, computer25 years

Senior form: Computer5 Alfred

Junior form: Biology, computer20 years

The five interviewees were three females and two males ranging in age from 28 to

47 years old and with teaching experience from 5 years to 25 years. Three of them were


47/97

40

specialised in teaching language subjects. They were secondary school teachers from

different teaching backgrounds and schools. Their schools were government-subsidised

except that one of them, Alfreds school, was a private one. In this study, anonymous

names were used to denote their identity as shown in Table 3.1.

3.5.4 The administration procedures

The whole interview exercise completed in six weeks. Since the five interviewees

were Cantonese speaking, for the sake of better interaction and communication, all the

interviews were conducted in Cantonese. During the interviews, questions were asked

in similar wordings, but with supplementary elaboration to interviewees when needed to

understanding of the questions in their context. The interviewer asked them some extra

open-ended questions to cover some important and salient topics that were inadvertently

omitted (Patton, 1980, p.206) and this offered opportunities for the interviewees to

provide in-depth qualitative feedback.

The first interview served as a trial run to sharp the interview questions to suit the

research objectives for the subsequent interviews. First, as Rachel was available, the

researcher carried out the first interview with her and spent much time to accomplish this

protracted interview. Having received the responses and suggestions by Rachel, some

questions were modified or omitted in order to collect more specified, befitting data in

outline form from the interviewees.

Nine preset, semi-structure questions were used in the five interviews. All

interviewees were well informed the purpose of this study beforehand. They focused on

the theme of the study to respond questions during the interviews. The researcher

transcribed their verbal responses as well as non-verbal responses including their factual

expressions and tones emanated through the interviews in such a way that richer data

could be collected. The researcher also provided feedback to the interviews to confirm


48/97

41

the validity of the transcription.

Moreover, the interviews were not merely a data collection exercise. All

interviewees were acquaintances of the researcher and the interviews were carried out

during coffee breaks, lunches, and dinners. The exercise was deemed to be social

encounters and the transcription was not simply a record of data. Other valuable

information emanated from non-verbal communication was recorded as well. Serving

interviews as social encounters helped avoid massive data loss, distortion and the

reduction of complexity (Cohen et al., 2003, p.281).

Next, the transcribed data were counterchecked by interviewees to minimise the

transcription errors and misinterpretation by the interviewer. Because of these, the

validity of this study could be enhanced in different ways. Mainly, the collected data

could accurately describe the situation and internal validity could be preserved.

3.5.5 Presentation of results

The interviewer tried to write down all the important verbal responses from the

interviewees including what was being said to answer the interview questions. Moreover,

their tone of voice to the questions was interpreted and transcribed by the interviewer.

Once the data had been transcribed from the interviews, the researcher employed

some tactics suggested by Miles and Huberman (1994) to generate meaning from the data.

Among the data, he counted the frequencies of occurrence of ideas, classified them into

categories, identified relations between variables, built a logical chain of evidence, and

made conceptual coherence to explain the situation.


49/97

42

CHAPTER 4: FINDINGS

4.1 Introduction

This Chapter sets out the findings from the questionnaire survey and the follow-up

interviews. By using the CBAM approach as elucidated in Chapter 3, the stages of

concern of teachers about the innovation of Standards-Refere

Documents

Impact of Prior Experience (Sample)