Embed Size (px)
Citation preview
A RESPONSE TO CHAPTER 6 OF THE DEPARTMENT OF
EDUCATION AND SKILLS DOCUMENT –
Better Literacy and Numeracy for Children and Young People:
A Draft National Plan to Improve Literacy and
Numeracy in Schools
Submitted by
The Educational Research Centre
St Patrick’s College
Dublin
February 28, 2011
ii
iii
Contents
About the Educational Research Centre iv
Preface vi
1 Introduction 1
2 Enhancing Teaching and Learning through Classroom
Assessments
5
3 Linking Classroom Assessments to Curriculum Standards 7
4 Standardised Testing at Four Key Points during Compulsory
Schooling
9
5 Contextualising the Outcomes of Standardised Test Results 13
6 National and International Assessments of Educational
Achievement
18
7 Conclusion 21
References 24
iv
About the Educational Research Centre
The Educational Research Centre was established on the campus of St Patrick's College,
Dublin in January, 1966. The setting up of the Centre was a means of widening the scope of,
and making for greater continuity in, research efforts in Ireland. The Centre works at all
levels of the education system, from pre-school to third level.
Research is undertaken by the Centre in the following areas:
Evaluations of Initiatives and New Programmes
The Centre conducts evaluations of initiatives and new programmes in schools to improve the
learning and life-chances of students in socially-disadvantaged situations. Completed projects
include evaluations of the Rutland Street Project, Breaking the Cycle, Early Start, and the
Home-School-Community Liaison Scheme. Current projects include an evaluation of School
Support Programme under DEIS (Delivering Equality of Opportunity in Schools).
Evaluations of Services and Existing Provision in Schools
The Centre conducts surveys of services, conditions and practice in schools. These include
topics as diverse as teachers‟ perceptions of girls and boys in primary schools, learning
support provision in primary schools, and dropout in third-level institutions.
Critical Analysis of Issues in Education
The Centre conducts research on a number of critical issues in education, including factors
associated with early school leaving. The designs of a number of Department of Education
and Skills schemes for addressing educational disadvantage have been influenced by studies
of the extent and geographical distribution of disadvantage (e.g., Giving Children an Even
Break, Delivering Equality of Opportunity in Schools). These studies also provided a basis
for assessing levels of disadvantage in individual schools and were used to select schools for
participation in the schemes.
Analysis of Public Examination Results
The Centre has on several occasions carried out analyses of the results of public examinations
(Junior Certificate and Leaving Certificate). Studies have been carried out on: (i) the impact
of examination components on total scores in the Leaving Certificate Examination; (ii)
relationships between performance on Certificate examinations and performance on
international assessments of educational achievement; (iii) the comparability of grades in the
Leaving Certificate Examination; and (iv) the relationship between performance on the Junior
and Leaving Certificate examinations.
National Assessments of Educational Achievement
As part of the Centre‟s work in monitoring the outcomes of education, national surveys of
achievement at primary level have been carried out in association with the Department of
Education and Skills. The sample-based surveys have been conducted in the areas of English
(reading), Irish (reading and oral language), and mathematics. In 2009, pupils in Second and
v
Sixth classes in primary schools were assessed on English and mathematics, while in 2010,
pupils at the same class levels in Irish-medium schools were assessed in these subjects.
International Assessments of Educational Achievement
The Centre has been involved in international assessments of educational achievement since
the 1980s. Current work includes the Organisation for Economic Co-operation and
Development‟s (OECD) Programme for International Student Assessment (PISA) in which
15-year olds in 34 OECD member countries, and several partner countries are assessed every
three years on reading literacy, mathematical literacy and scientific literacy. Current work
also includes two studies organised by the International Association for the Evaluation of
Educational Achievement (IEA) – the Progress in International Reading Literacy Survey
(PIRLS) and the Trends in International Mathematics and Science Study (TIMSS) – both of
which involve pupils in Fourth class in primary schools.
Development of Assessment Instruments
Work is carried out on an ongoing basis on the development of tests and profiling systems
which teachers can use to assess pupils in primary and post-primary schools. These include
the Drumcondra English Profiles, the Drumcondra Primary Reading Test, the Drumcondra
Primary Mathematics Test, the Drumcondra Reasoning Test, and Trial Ghaeilge Dhroim
Conrach do Scoileanna Lán-Ghaeilge agus Scoileanna Gaeltachta. Most recently, the Centre
has published sets of screening and diagnostics tests for children in Senior Infants and First
classes in primary schools – the Drumcondra Tests of Early Reading Literacy and the
Drumcondra Tests of Early Numeracy. The Centre has recently begun work on the
development of standardised tests in English and mathematics for Second-level schools.
vi
Preface
The Educational Research Centre welcomes the publication by the DES of the Better Literacy
and Numeracy for Children and Young People: A Draft National Plan to Improve Literacy
and Numeracy in Schools.
The results of PISA 2009, which were published in December 2010, have received a lot of
attention and provoked some negative comment about education in Ireland. While it is
important to be cautious about PISA 2009 findings relating to declines in performance (see
Cosgrove et al., 2010; Perkins et al., 2010; LaRoche & Cartwright, 2010), it is clear from all
PISA cycles, and from other assessments including National Assessments of English reading
and mathematics conducted over many years, that a range of issues relating to performance in
reading and mathematics need to be addressed. These include the poor performance of
students in mathematics at both primary and post-primary levels, with mathematical problem
solving being an area of particular concern. They also include the relatively poor performance
of higher-achieving students in Ireland, who do not do as well as might be expected in either
reading literacy or mathematics. PISA 2009 also draws attention to the challenges facing
students for whom English or Irish is not the language of the home.
The Draft Plan is a welcome attempt to address these and other issues in the educational
system that impact in a negative way on students‟ performance and, by inference, their life
chances.
In framing our response to the Plan, we have focused on Chapter 6, which presents detailed
proposals for assessment. We hope that our comments will be considered by the DES as it
develops its strategy to improve student performance, and we look forward to co-operating
with the DES in its efforts to ensure that students in Ireland achieve their potential in all
aspects of the curriculum, including literacy and numeracy.
1
1. Introduction
In this section, we describe the different types of assessments referred to in Chapter 6 of the
Draft Plan. We define each assessment type, summarise its purpose, and link it to the
corresponding proposals in the Draft Plan (see Table 1-1). We then identify the section of this
response where each assessment will be considered in greater detail.
Table 1-1
Types of Assessments Referred to in the Draft Plan
Type of Assessment Description Proposals in Draft Plan
Classroom Assessments Based on
Curriculum Standards
Medium-stakes assessment for
schools in which teachers rate or
grade student achievement in L1,
L2 and mathematics with reference
to national standards, using
exemplars, scoring rubrics, or other
tasks.
Assessment to be implemented at
five stages during compulsory
schooling. Results to be shared at
school level, and with Boards of
Management.
Schools Like Ours Standardised
Testing
High-stakes assessment of English
reading and mathematics for
schools in which they administer
standardised tests and submit
results to a central unit, for
comparison with other schools
sharing similar intake
characteristics.
Tests to be administered at four
points in compulsory schooling.
Results to be used at school level
for monitoring and target setting,
presented to Board of
Management, and shared with
Inspectorate. Aggregated national
results to be published.
National Assessments Low-stakes assessments of English
reading, mathematics (and
sometimes Irish) for schools and
students. Feedback on each
school‟s performance provided to
school only; system level
performance published.
Outcomes to be used to monitor
achievement of national targets
specified in the Draft Plan.
Currently administered in Second
and Sixth classes every four years.
To be extended to Fourth class
(Primary) and Second year (Post-
primary).
International Assessments Low-stakes for schools; High-
stakes for the system. Performance
reported at national and
international levels.
Participation in international
assessments to continue. Ireland
currently participates in three such
assessments – PIRLS reading
literacy (4th
class), TIMSS maths
and science (4th
), and PISA (15-
year olds attending post-primary
schools).
Classroom Assessments Based on Curriculum Standards
Currently, teachers are not required to assess student performance with reference to national
standards. However, First Steps (which is used in many schools in the SSP under DEIS)
includes an assessment component that is based on teacher judgements about student success
on key learning outcomes presented along a developmental continuum.
2
Assessment of student performance according to national curriculum standards would
represent a significant challenge for schools and teachers, since, as noted above, there is no
current requirement for this type of assessment. In 2000, the Educational Research Centre
published a framework for standards-based assessment, the Drumcondra English Profiles
(Shiel & Murphy, 2000). The Profiles cover oral language, reading and writing, and can be
used to assess pupils in primary schools (Junior Infants to Sixth class) against key curriculum
outcomes. Although the Profiles were widely disseminated, subsequent surveys of teachers‟
assessment practices suggested modest uptake (e.g., Eivers et al., 2005, where teachers of
54% of pupils in First class reported that they never used curriculum profiles). However,
usage levels need to be considered in the context of concerns raised by the Irish National
Teachers Organisation soon after dissemination about the time required by teachers to
implement the Profiles. Profiles for Irish-medium schools were published by the Centre in
2004 (O‟Siaghail & Déiseach, 2004).
In Table 1-1, we have categorised classroom assessments based on national standards as
„medium stakes‟. This is because the assessments may have some consequences for schools,
if, as proposed in the Draft Plan, schools are required to report the outcomes to parents and
school management.
We return to teacher assessments based on curriculum standards in later sections of this
response. In Section 2, we consider the use of classroom-based assessments for formative
purposes, without specific reference to curriculum-based national standards. In Section 3, we
look at classroom-based assessments linked to standards. We believe that classroom
assessments with and without curriculum standards need to be developed and implemented in
classrooms.
Standardised Tests
Norm-referenced standardised tests are widely used in primary schools. According to Eivers
et al. (2010), standardised tests of reading are administered to 95% of pupils in Second class
and to 88% in Sixth, while standardised tests of mathematics are administered to 95% of
pupils in Second and 90% in Sixth. Part of the impetus for this comes from Circular
0138/2006 (DES, 2006), which requires schools to administer standardised tests at two points
during primary school: the end of First class or the beginning of Second class, and the end of
Fourth class or the beginning of Fifth class. However, the high levels of usage at other class
levels also reflect an acceptance of standardised tests by most schools, at least for the
purposes for which they are currently used.
The Draft Plan proposes that standardised tests would be administered at four points:
Second, Fourth and Sixth classes (primary) and Second year (post-primary). The requirement
to administer standardised tests in the Second year of post-primary schooling appears to
reflect increased concerns about literacy and numeracy standards at post-primary level. Such
concerns arises despite the fact that just 1.7% of candidates achieved a grade E or lower in
Junior Certificate English in 2010, and 5.6% achieved a similar level in Mathematics (State
3
Examinations Commission, 2010a)1. However, in 2009, 17% of 15-year olds in post-primary
schools achieved at or below Level 1 on PISA reading literacy, indicating poor performance
in that domain, and 21% had a similarly low performance on PISA mathematics (Perkins et
al., 2010).
We have labelled the proposed standardised testing programme in the Draft Plan as Schools
Like Ours Standardised Testing. This is because it is proposed to gather test data from
schools and to use the data to enable schools to benchmark their average performance against
other schools with similar characteristics. We have categorised Schools Like Ours
Standardised Testing as „high stakes‟ because aggregated test results, and information about
performance targets based on the results, are to be reported to school management, and test
results are to be used by members of the Inspectorate to identify schools that are performing
above or below expectations (Draft Plan, p. 46).
In a recent report prepared for the NCCA, Shiel, Kellaghan and Moran (2010) examined a
number of options for the introduction of standardised tests of literacy and numeracy in post-
primary schools. Two key options were put forward:
Standardised tests of achievement in literacy (English/Irish) and numeracy with Irish
norms developed for the three years of lower secondary schooling, and made
available to schools to be administered when they consider it appropriate, to support
monitoring of student progress (the „Netherlands model‟)
Standardised tests developed and mandated for use at one point in lower-secondary
schooling, such as the first term of first year („the French model‟, but without central
reporting), or at the end of Second/beginning of Third year, when results might be
used for guidance purposes [within schools] (p. 198).
Significantly, the options did not include census-style every-pupil testing where results would
be submitted to a central body for further analysis and comparison with other schools.
However, it was noted that the issue of whether the results of standardised tests should be
available to the DES (in summary form), as occurs at primary level in the context of Whole
School Evaluation, might arise.
In this response, we consider the implementation of standardised testing from two
perspectives. In Section 4, we look at issues arising from the extension of standardised testing
to Fourth class (primary) and Second year (post primary) without reference to the Schools
Like Ours initiative. In Section 5, we look more closely at implications of the possible
introduction of Schools Like Ours, including the proposal to compare performance across
schools with similar student intake, and to „report on the national achievement trends in
literacy and numeracy at each of the four stages using Schools Like Ours data‟ (p. 46).
1 The corresponding percentages at Leaving Certificate level in 2010 were 2.4% for English and 8.3% for
mathematics (State Examinations Commission, 2010b)
4
National and International Assessments
The Draft Plan proposes that national assessments of achievement in English reading and
mathematics, which are currently administered in Second and Sixth classes in primary
schools, be expanded to include Fourth class (primary) and Second year (post-primary). The
national assessments can be categorised as „low stakes‟ because there are no consequences
for schools, teachers or students. As discussed in Section 6, national assessments can provide
useful information on performance levels, as well as on the performance of students in key
subgroups. Recent national assessments have also seen the development of proficiency levels
in English and mathematics, and these may be relevant for other forms of assessment
proposed in the Draft Plan such as standards-based classroom assessments.
The Draft Plan refers to Ireland‟s continued participation in international studies of reading
literacy and mathematics. Currently, Ireland participates in three such studies – two at
primary level (PIRLS and TIMSS) and one at post-primary level (PISA). Section 6 addresses
how these assessments might be integrated with current national assessments to provide a
balanced picture of performance and instructional practices across schools.
The concluding section of our response (Section 7) considers the proposals for assessment of
literacy and numeracy in the Draft Plan in their totality and identifies some possible
implications for the educational system if all of them are implemented within the proposed
timescale. In the same section, we propose an alternative timeline that allows for some of the
assessment proposals to be examined in more detail before they are implemented across all
schools in the system.
5
2. Enhancing Teaching and Learning through Classroom Assessments
We welcome the emphasis on classroom-based assessment (also termed assessment for
learning or AfL) in the Draft Plan. We agree that “gathering evidence about how well
students are learning and using this information to improve learning opportunities” is an
essential element of literacy and numeracy development (Draft Plan, p. 39). We also agree
that „good practice in assessment means using a variety of assessment methods in order to
provide a full picture of a learner‟s achievement‟ (ibid, p. 39). Indeed, we would point to the
NCCA document, Assessment in the Primary School Curriculum – Guidelines for Schools
(NCCA, 2007) as an important source that outlines strategies for integrating assessment into
teaching and learning, including strategies for assessing aspects of literacy and numeracy.
It is important, however, to draw a clear distinction between the assessments that teachers
conduct on a day-to-day basis to support teaching and learning in the classroom and the
standards-based assessment of learning proposed in the Draft Plan. Classroom-based
assessments, which teachers administer on an ongoing basis, may include:
Sharing goals, expectations, targets and success criteria with students
Analysing student talk (discourse)
Evaluating students‟ mastery of objectives
Identifying students‟ errors
Assessing students‟ prior knowledge and knowledge structures
Assessing students‟ problem-solving abilities (linguistic, schematic, strategic and
algorithmic knowledge)
Questioning students to explore their knowledge, understanding and misconceptions
Evaluating critical thinking
Providing feedback – whether norm-referenced, criterion-referenced or self-
referenced
Developing students‟ self assessment capacity
Promoting peer-assessment.
The outcomes of classroom-based assessments can be used by teachers on a daily basis to
inform teaching and learning without accountability consequences for teachers or students.
There is strong support in the research on assessment for classroom-based assessments or
AfL (Black & Wiliam, 1998), and many benefits are associated with AfL. According to
Afflerback and Cho (2011), they include the following:
AfL can contribute to enhanced student learning, as teachers accurately identify
students‟ learning strengths and needs, and teach within students‟ zones of proximal
development.
AfL conducted in classrooms can take into account the situated nature of learning
(i.e., the specific contexts in which teaching and learning occur, including students‟
states of affective and emotional development).
AfL can accommodate broad definitions of literacy (and numeracy) that include
cognitive as well as non-cognitive outcomes of instruction (self-concept, self-esteem,
attributions of performance, locus of control, motivation and attitude).
6
AfL can provide teachers with information that enhances their role as „reflective
practitioners‟ which, in turn, can contribute to better-informed teaching.
AfL can improve students‟ ability to assess their own learning through the
enhancement of their megacognitive skills. Access to the scoring rubrics that teachers
use to rate performance assessments can inform students about the criteria against
which they will be assessed, increasing the likelihood that they will internalise those
criteria (i.e., self-assess).
AfL can modify the socio-cultural space of classroom discourse. Teacher discourse
patterns may change as teachers focus on higher-level thinking and problem solving,
since traditional discourse patterns may not support assessment of such learning.
AfL could enable teachers to focus assessment efforts more fruitfully on students with
the greatest needs (e.g., children with learning difficulties, children from
disadvantaged backgrounds, English language learners).
Specific strategies for classroom-based assessments have been outlined in detail in
Assessment in the Primary School Curriculum – Guidelines for Schools (NCCA, 2007).
These include self-assessment, portfolio assessment, questioning, teacher observation etc.
These assessments have obvious implications for teachers‟ continuous professional
development (CPD), and should be included in CPD on literacy and numeracy (as proposed
in Chapter 2 of the Draft Plan). Indeed, an initial focus on classroom-based assessment in
CPD might lay a foundation for future work involving standards-based assessment as the two
types of assessment share several features. The immediate introduction of standards-based
classroom assessments, without first establishing a strong underlying knowledge base about
classroom assessment strategies among teachers, may not be successful.
7
3. Linking Classroom Assessments to Curriculum Standards
The Draft Plan includes specific proposals for the development of standards-based
assessment. It is envisaged that this will consist of two steps – developing standards based on
the curriculum, and providing supports to teachers (e.g., exemplars of performance at
different levels of achievement, Internet-based item banks) so that they can rate their
students‟ achievements against the standards (page 44). It is further envisaged that this work
will occur on a formal basis in Senior Infants, Second, Fourth and Sixth classes at primary
level, and in Second year at post-primary level. There is an expectation that teachers can
„readily, validly and reliably interpret and communicate their students‟ progress in achieving
national standards‟ (page 44).
We welcome the proposal to establish curriculum-based standards for student learning in the
context of classroom assessments and see it as something that builds on our earlier work on
the development of a structured programme for the infant classes (Gleeson, Kelly & Archer,
1988a, 1988b, 1988c) and the Drumcondra English Profiles (Shiel & Murphy, 2000).
A number of issues arise when considering the development of standards:
It is envisaged that national standards for English at primary level will have been
completed by the end of 2012 (p. 44). However, on page 29 there is a target for
review of curriculum content of L1 (English or Irish) at primary level and the
specification of learning outcomes is to be completed in the 2012-13 school year.
Clarification is needed on whether learning outcomes and national standards are one
and the same. If it is envisaged that national standards will be based on learning
outcomes, there may be a sequencing problem.
It would seem important to revise the content of L1 curricula before developing
national standards. The current L1 and mathematics curricula at primary level have
now been in place for over a decade, implementation of these curricula has been
evaluated, and information is available on their strengths and weaknesses, through
evaluation reports (e.g., NCCA, 2005, 2008; DES, 2005), and reports on national
assessments (Eivers et al., 2005; Eivers et al., 2010). The DES Inspectorate has
looked in detail at the teaching of English at post-primary level (DES, no date).
Can standards be derived from proficiency levels already established in the context of
existing assessments of English reading and mathematics? Considerable work has
already been done on the development of proficiency levels for English reading and
mathematics for Second and Sixth classes in the context of national assessments
(Eivers et al., 2010), while proficiency levels are already available for mathematics in
Fourth class (Shiel et al., 2006), and for standardised tests of mathematics that are
available to schools (Educational Research Centre, 2008). It would seem useful to
explore possible alignment between national standards and the proficiency levels that
have already been developed. The approach adopted in England has been to link
teacher assessments and standardised tests to the same underlying set of level
statements (standards). Is it envisaged to do the same here? If two different sets of
8
standards are in use (curriculum-based standards and proficiency levels based on
current curriculum-based national assessments), there is potential for confusion.
If standards are established independently of students‟ current levels of achievement,
there is a risk that the standards will be too difficult to achieve (i.e., large numbers of
students will not reach them), or too easy (almost all students will reach the highest
standards for their class level).
Moderating Classroom Assessments Based on Standards
As noted earlier, the standards-based teacher assessments proposed in the Draft Plan are
summative in nature and are medium-stakes to the extent that the results are to be shared at
school level and reported to a school‟s Board of Management. In this context, it would seem
important that a system of moderation be put in place, at least within schools or clusters of
schools, so that high levels of consistency can be achieved in rating students‟ performance.
While support materials are to be provided – exemplars representing different levels of
achievement, and a bank of Internet-based assessment resources – there will still be a need to
ensure that teachers within and across schools interpret the standards in similar ways.
It can be argued that a system of moderating teachers‟ judgements through professional
collaboration benefits teaching and learning as well as assessment (Assessment Reform
Group, 2006). Such moderation might involve planning assessments, collecting relevant data,
and analysing the data in a collaborative setting. This, in turn, could enhance teachers‟
understanding of learning goals and criteria indicating progress towards them.
9
4. Standardised Testing at Four Key Points during Compulsory
Schooling
In this section, we consider the proposed extension of standardised testing in schools, without
reference to Schools Like Ours, which is looked at in the next section. Here, the focus is on
implications of the Draft Plan for the expanded use of standardised tests within schools.
Standardised tests can serve a variety of functions. These range from evaluating the effects of
new interventions to monitoring standards across the educational system. However, it is often
difficult and sometimes impossible for the same testing programme to serve multiple
functions effectively. Standardised tests that aim to provide teachers with information on the
performance of individual students in their classes may not be suitable for monitoring
standards across schools, or evaluating the educational system.
Use of Tests within Schools
Proposals in the Draft Plan call for the extension of standardised testing in L12 and
mathematics to four points during compulsory schooling – the end of the Second, Fourth and
Sixth classes at primary level, and the end of Second Year at post-primary level – using
standardised achievement tests of literacy and numeracy (p. 45). They also refer to a
requirement for schools to report outcomes for individual students to parents and for
principals to report aggregated data to Boards of Management on an annual basis (p. 45).
CPD is to be provided to schools on how to use the outcomes to support school self-
evaluation and target setting (p. 45).
In general, these proposals are uncontroversial. Primary schools are currently required to
administer and report to parents on the outcomes of standardised tests at the end of First class
or the beginning of Second class, and at the end of Fourth class or the beginning of Fifth class
(Circular 0136/2006), while target setting is a component of planning for schools in the
Schools Support Programme under DEIS. A shift in standardised testing to the end of
Second, Fourth and Sixth classes should be accomplished without difficulty since schools
already administer standardised tests at most class levels. According to Eivers et al. (2010),
standardised tests of reading are administered to 95% of pupils in Second class and to 88% in
Sixth, while standardised tests of mathematics are administered to 95% of pupils in Second
and 90% in Sixth.
There are issues about reporting test results to Boards of Management in the case of small
schools (mainly rural primary schools), where there may be just a handful of pupils
(sometimes as few as one) enrolled at a class level. In such cases, it would be problematic to
issue standardised test result such as the percentages of students at a class level performing at
each of several proficiency levels, since the identity of pupils may be compromised.
2 We assume the Draft Plan intends that standardised testing will involve L1, whether English or Irish. While
reference is made to L1 in the context of classroom assessments based on curriculum standards, the situation is
less clear in respect of standardised tests.
10
Testing at Post-primary Level
There are no standardised tests of reading or mathematics with current national norms
available at post-primary level. Hence, it will be necessary to develop such tests, which
should be curriculum-related3. The indicative date in the Draft Plan for implementation of
standardised testing in post-primary schools (2011-12) seems over-ambitious in this respect,
as high-quality standardised tests for Second year would take two to three years to develop.
Furthermore, since a new mathematics course is currently being implemented at Junior Cycle
level (Project Maths), it may take two to three years before the content and focus of the new
syllabus is fully understood (with the exception of pilot schools, 2011-12 will be the first year
in which Project Maths will be taught in Second year). Hence, the ERC would suggest
delaying implementation of standardised testing at Second Year until at least 2013-14.
According to the Draft Plan, standardised tests at post-primary level are to be connected to
the outcomes of the Junior Cycle syllabus. However, it is unclear if key outcomes of Junior
Certificate English, especially those dealing with higher levels of text interpretation and
evaluation, can be assessed using standardised tests where, for reasons of cost and efficiency,
many if not most items must be in multiple-choice format. Perhaps clarity about this issue
will be achieved, when, as proposed in the Draft Plan, the current Junior Cycle English
syllabus is revised and specific curriculum outcomes are identified. However, it is as yet
unclear how proposed syllabus revisions will align with the overall review of the Junior
Certificate currently being conducted by the NCCA.
Age vs. Grade Norms
The Draft Plan states that scores of individual children on standardised tests should be
reported to parents with reference to „the norm for children of their age‟ (p. 45). Since it is
specified that tests would be administered at particular class levels, this suggests that
performance would be reported with respect to age norms within class levels. This would
require separate norms for each age range within a class level and, in some cases, a
renorming of existing tests. There is little obvious benefit to such a change, and the ERC
recommends that reference to age-based norms be replaced by a reference to grade-based
norms.
Target Setting
The proposal that schools would engage in self-evaluation and set targets for improvement
based on standardised tests needs some refinement:
Many primary schools are small, and numbers in Second, Fourth and Sixth classes
fluctuate dramatically from year to year. Furthermore, even if numbers stay stable
(e.g., 8 or 9 students at a grade level), the characteristics of the students may fluctuate
from year to year (different socio-economic composition; different gender
composition; different proportion of ESL students and so on), limiting the inferences
3 Some preliminary work on the development of standardised tests began at the Educational Research Centre in
October 2010.
11
that schools, teachers and Boards of Management could make, using average scores or
the percentages of students at each proficiency level. In Ontario, Canada, school-
level standardised test scores are not reported outside of schools if there are fewer
than 15 students enrolled at the grade level tested. In Ireland, this would exclude
almost 60% of primary schools.4 In the case of small schools, it may be more useful to
establish targets in respect of individual students rather than class-level cohorts.
What targets are realistic? At this time, we don‟t know how much progress the
highest-performing schools can reasonably make from year to year. Thomas, Peng
and Gray (2007) examined a GCSE data-set over a period of ten cohorts (1993-2002)
and found that only one in 16 schools managed to improve continuously for more than
four years at some point over the decade in terms of value added. Also, they found
that schools with a lower value-added starting point in 1993 were more likely to make
significant improvement.
It would seem important, as a prerequisite to providing CPD to schools on target
setting, that some research be undertaken on what schools can reasonably expect to
achieve in terms of raising scores from year to year relative to their intake, the
resources that are available to them, and whether schools can be expected to make
similar levels of improvement over several years.
The targets for literacy and numeracy on page 12 of the Draft Plan (a 5% increase at
each of Levels 3 and 4 on the National Assessments of English and Mathematics at
Second and Sixth classes between now and 2020, and a reduction of 5% in those
scoring at or below Level 1)5 seem reasonable, through it should be noted that,
between 1980 and 2004, no change was observed in overall performance in English
reading at Fifth class level (Eivers et al., 2005). Similarly, no increase in overall
performance was recorded for mathematics in Fourth class between 1999 and 2004,
despite the introduction of the revised Primary School Mathematics Curriculum
during that time (Shiel et al., 2006). At this point it is unclear what the targets mean
for individual schools, especially those schools where student numbers are small, and
performance might be expected to fluctuate from year to year.
The experience gained in setting and maintaining targets in DEIS schools should prove
helpful in achieving the refinements of the proposals in the Draft Plan that we see as
necessary.
4 In the UK, results for schools with 10 or fewer pupils are not published in league tables of GSCE results. Ray
(2006) states that "there is no easy solution to the problem of interpreting value added for small schools" but
suggests that one possibility is to combine results across years. 5 It would be important to provide a justification in the Plan for these targets. For example, why was 5% selected
rather than 10%?
12
Provision of Diagnostic Information
We would encourage the provision of diagnostic information to teachers – i.e., information
on the strengths and weaknesses of individual pupils – in the context of reporting
standardised test results. In their work on the effects of standardised tests in low-stakes
contexts, Kellaghan et al. (1980) found that more progress was made by students of teachers
who received both norm-referenced and diagnostic test results in respect of individual
students, compared with students of teachers who received norm-referenced results only, or
no results at all. With increased access to computer scoring of standardised test performance,
it should be possible to provide more detailed diagnostic information, at school and student
levels, than has been possible in the past. While schools and teachers may find that the
descriptions of achievement embedded in the proficiency levels that accompany some
standardised tests are useful in terms of identifying which tasks or processes a student is
likely to be successful on, there may be a case for generating more detailed, item-level
information as well.
We also recognise the use of tests specifically developed to serve diagnostic purposes.
Indeed, the ERC has recently completed work on the development of diagnostic tests of early
literacy and numeracy for at-risk students in Senior Infants and First class (ERC, 2010,
2011).
Potential of Computer-based Assessment
We recommend that consideration be given to assessing reading and mathematics using a
computer-based system, particularly at Sixth class and Second year. This would greatly ease
the test administration burden on schools, and have the potential to facilitate adaptive testing,
where most of the items that students are asked to respond to are at a level at which they have
a reasonable chance of being successful. This can also reduce the time required for testing
and produce more accurate scores as students would not be expected to answer questions that
are too easy or too difficult for them (something that happens when current print-based tests
are administered).
In PISA 2009, Ireland was among 19 countries that participated in a computer-based
assessment of reading. The outcomes of this (to be released in 2011) should be instructive in
illuminating similarities and differences in student performance in paper-based and electronic
models, and inform decisions on how best to proceed with computer-based assessment of
reading in the Irish context. It might also be pointed out that, over time, PISA hopes to move
fully to computer-based assessment – something that has already been done in the case of
national assessments in Denmark (see Wandall, 2008).
13
5. Contextualising the Outcomes of Standardised Test Results
The Draft Plan contains a number of proposals relating to the aggregation of standardised test
results generated in schools. The proposals include collection by an external agency of
standardised test results for all pupils at the end of Second, Fourth and Sixth classes in
primary schools, and the end of Second year in post-primary schools (p. 45). It is proposed
that schools would then benchmark their performance against other schools with similar
background characteristics, using online tools. Schools would be required to report on
„comparative performance of the school‟s students against national trends as well as
standards in schools in similar contexts‟ (p. 45). The outcomes would also be used to „report
on national achievement trends at each of four stages‟ (p. 45).
There are a number of issues that should be taken into account in considering how this
proposal might operate in practice:
In the case of English reading and mathematics, schools currently use a number of
nationally-normed standardised tests. There are obvious difficulties in comparing the
performance of schools using different tests. For example, although two tests may be
nationally-normed, one may be easier or more difficult than the other at the outset or
may become easier or more difficult over time, giving an unfair advance to some
schools and inadvertently penalising others. In the same vein, high (or low) achievers
may be at an advantage on one test, compared with the other.
Although statistical linkages may be established between two or more tests by
administering them to the same set of students and placing performance on a common
underlying scale, such a process would inevitably result in an increase in the error
associated with individual and aggregated test scores, making changes in performance
more difficult to detect. It is worth noting here that one underlying test only is used in
each target grade in National Curriculum Assessment in England, NAPLAN (National
Assessment Program – Literacy and Numeracy) in Australia and No Child Left
Behind at state level in the US, with some content changing from year to year.
Schools and teachers may become over-familiar with current standardised tests of
English reading and mathematics in the context of high-stakes testing (with
aggregated results to be reported to a central agency and to school management).
When familiarity with a test increases, teachers may inadvertently teach to the test, at
the expense of teaching well. In such circumstances, increases in performance may
not reflect actual changes in underlying proficiency.
14
Continuous vs. Categorical Social Context Measures
We have a number of concerns about the proposal to cluster schools by category (for
example, DEIS Band 2 urban schools where English is not the first language of 25% of
students – Draft Plan, p. 41):
Assuming that a measure based on average socioeconomic status (such as DEIS
status) is used to categorise schools, those schools with the lowest SES levels will
find themselves at the bottom of a category, and will inevitably find it more difficult
to achieve average performance for the category, compared with schools that are
closer to the middle of the category, or those near the top. This could have a negative
effect on morale in lower-SES schools in a category, especially if they are viewed as
performing below expectations. Another potential difficulty is that, if too crude a
measure of school socioeconomic status is used, schools of differing socioeconomic
status could be incorrectly grouped together.
The Draft Plan does not consider the use of continuous variables to adjust school
average achievement scores (e.g., an updated version of the measure used to identify
schools in terms of levels of disadvantage in the lead-in to DEIS in 2005). If a
continuous measure of school socio-economic status were available, it and other
relevant variables could be used to adjust school average scores. The ERC believes
that adjusting average scores by modelling achievement using a range of relevant
school and student-level background variables is a more accurate and reliable method
than that proposed for Schools Like Ours.
The Draft Plan does not consider the possibility of comparing the performance of
schools over time, using a relevant pre-test measure (and hence implementing a value-
added approach). There is now a significant and growing body of research on
approaches to contextualising value added change in schools (e.g., OECD, 2008) that
might be applicable to the Irish situation if suitable and current measures of school
and student background were available for all schools and students. The more fine-
grained information that would derive from value-added growth modelling could be
more useful to schools than the less sensitive category-based information that would
emerge from Schools Like Ours. 6
However, the complexity of value-added modelling
should not be overlooked either (e.g., Braun, Chudowsky & Koenig, 2010; Ray,
2006).
We return to the matter of value-added models of growth in Section 7.
Frequency of Gathering Aggregated Test Data
The Draft Plan specifies that school-level test data (and possibly contextual data) will be
gathered on an annual basis. Gathering data on such a frequent basis may result in schools
6 For example, multi-level modelling, in the context of value-added analysis, allows for an examination of
whether schools are differentially effective for different subgroups of students.
15
allocating undue amounts to time to administering and scoring tests, and sourcing
background information on students. This is time that could be allocated to other aspects of
teaching and learning, including classroom assessments (whether or not they are linked to
curriculum standards). A longer interval between data collection points (e.g., three years)
would also reduce the resource demands on schools, and perhaps sustain the testing
programme over a greater number of years. It is relevant in this regard that the Draft Plan
suggests that schools be required to establish three-year plans to improve literacy and
numeracy.
The Situation of Small Schools
As noted earlier, the situation of small schools needs to be examined carefully. In some small
schools, numbers of students are likely to fluctuate substantially from year to year, while in
others, the social composition of a cohort may fluctuate. It would be unsafe to report average
performance, progress towards reaching targets, or comparative performance for such
schools. Moreover, the challenge presented by having a large number of small schools in the
system would arise in model-based approaches to tracking performance changes in the same
way that it would arise in Schools Like Ours.
A related issue concerns the treatment of students who enrol in a school during the year of
testing. Should they be included in school average scores, given that their prior schooling
occurred elsewhere, and, especially in the case of smaller schools, could they have a
disproportionate influence on average scores?
Exemptions from Testing and Provision of Support to Students with Disabilities
The Draft Plan is unclear on who will be tested, and who will be exempted from testing (i.e.,
excluded from aggregate scores submitted by schools). A clear policy on this issue needs to
be developed, as does a policy on dealing with students who are absent on the date of testing.
At post-primary level, between 80% and 85% of students selected to participate in the OECD
Programme for International Student Assessment (PISA) in Ireland present themselves on the
day of the test. The vast majority of non-presenting students are absent on the day of testing
and we know that on average their achievement on English, mathematics and (where
applicable) science on the Junior Certificate examination is lower than that of students who
present. At primary level, the situation is somewhat better, with participation in national
assessments generally in excess of 90% (Eivers et al., 2010, Table 2.9). Many testing
programmes similar to that proposed in the Draft Plan require a participation rate in excess of
95% so that results across schools are directly comparable. In some programmes (e.g., No
Child Left Behind in US states), there is a requirement to assess students with learning
disabilities and students for whom English is not the language spoken at home.
Related to the exemptions issue is whether second-level students with severe learning
difficulties are to be provided with the same types of supports they can access when they take
the Junior Certificate Examinations. Will schools have discretion in determining what levels
of support are to be given? If so, how can we know that there is consistency from school to
school, and over time?
16
Extent of Deviation of Schools from Expected Performance Levels
Proposals for Schools Like Ours are based on identifying schools that display average
achievement that is significantly above or below what would be predicted on the basis of
their socioeconomic composition. Currently, we cannot quantify this phenomenon. An
analysis completed in the context of the 2009 National Assessment of Mathematics and
Reading (Eivers et al., 2010) shows that, when error bands are established around a fit line
defining the relationship between socioeconomic status and reading, the majority of schools
fall within the error bands, with just a handful of schools outside. Further work could be
undertaken using existing datasets (Junior Certificate grades, DEIS data) to establish the
extent to which schools deviate from expected performance, based on socioeconomic status
or other intake factors. The following questions could inform the research:
What proportion of schools „underperform‟ or „over-perform‟ in reading literacy and
mathematics, after adjusting for the socioeconomic status of the school?
Do these schools consistently under- or over-perform in different areas (subjects) and
over time?
What are the characteristics of „under-performing‟ and „over-performing‟ schools7?
Use of Schools Like Ours Data to Report on National Trends
Finally, we would caution against using Schools Like Ours standardised test results to report
on achievement trends in literacy and numeracy, as suggested on page 46 of the Draft Plan.
First, the national assessments (described in Section 6) are intended to report on achievement
trends in these important areas. Current national assessments undertaken by the ERC share
the following characteristics:
Detailed assessment frameworks
Test items designed to assess a broad range of skills
A balance between multiple-choice and constructed response items
Clear rules on exclusion of students from testing
Monitored administration of the tests in schools
Use of secure tests that are not available to schools
Translation of test materials into Gaeilge
Controlled scoring of constructed response items to ensure reliability.
Use of complex statistical procedures to link test performance over time.
Extensive quality control including independent review of statistical analyses by an
international consultant
Support in interpreting outcomes from a national advisory group
Schools Like Ours standardised testing would not have any of these features. Therefore, it
would be unsafe to publish aggregated national outcomes, or to use those outcomes to draw
inferences about national performance standards.
7 An exercise of the type described here in relation to the population of schools in general is planned as the next
phase in the evaluation of the SSP under DEIS, focusing on schools in that programme.
17
Impact of Whole Cohort Testing
In thinking about the scale of the testing programme implied by Schools Like Ours, it is
instructive to note that there has been a growing realisation in England that full-cohort testing
(such as that proposed under Schools Like Ours) represents a significant burden to the
educational system (House of Commons Select Committee on Children, Schools and
Families, 2008). Indeed, full-cohort testing no longer applies to the end of Key Stage 3 (age
14) in England. Instead, schools now submit the results of moderated classroom-based
assessments to a central unit.
18
6. National and International Assessments of Educational Achievement
The Educational Research Centre has been involved in implementing national assessments of
educational achievement on behalf of the Department of Education and Science since the
early 1980s. In 2009, national assessments in English and mathematics were administered to
representative national samples of pupils in Second and Sixth classes, with results reported in
November 2010 (see Eivers et al., 2010). In 2010, the 2009 assessments were administered
to representative samples of students in Scoileanna Lán-Ghaeilge and in Gaeltacht schools. A
report on this study will be ready later in 2011.
In general, national assessments are used to monitor the performance of the educational
system as a whole. This is reflected in the aims of the 2009 national assessments:
To establish current reading and mathematics standards of Second and Sixth class
pupils;
To provide high-quality and reliable data to assist the DES in policy review and
formulation, and in decisions regarding resource allocation;
To examine school, teacher and home background and pupil characteristics, and
teaching methods which may be related to reading and mathematics achievement;
To provide a basis against which to compare outcomes of future national assessments
of English reading and mathematics at Second and Sixth classes (Eivers et al., 2010,
p. 1).
Since 2009 was the first year in which students were assessed in Second and Sixth classes, it
was not possible to interpret the outcomes with respect to trend data. However, a structure
has been established so that future national assessments can link back to performance in
2009.
According to the Draft Plan, it is proposed to continue with national assessments of reading
and mathematics in a representative sample of schools on a four-yearly basis (p. 46). It is also
proposed to extend the national assessments to Fourth class at primary level and Second year
at post-primary level (p. 12). If this occurs, there will be two parallel testing programmes
designed to gather national level data. In one, Schools Like Ours. aggregated data based on
standardised tests administered by teachers would be collected and submitted to a contractor
of the DES in respect of all pupils in the Second, Fourth and Sixth classes at primary level,
and Second year at post-primary level. In the other (National Assessments), broadly similar
data would be generated at the same class levels every four years, albeit in a sample of about
150 schools rather than across the population of students at these class levels (Table 6-1).
There is a danger that schools might not be able to accommodate the two testing programmes
on those years in which they overlap (every four years according to the Draft Plan). Since one
(Schools Like Ours) is a high-stakes testing programme, with results to be reported to Boards
of Management, and to members of the inspectorate engaged in school evaluation, and the
other (the National Assessments) is low stakes, schools may be under too much pressure to
19
engage fully with the latter. Again, the frequency of Schools Like Ours testing might be
reconsidered to ensure that schools are not over-burdened, and have the time and energy to
engage with other assessments as well.
Table 6-1: Comparison of Schools Like Ours Standardised Testing and
National Assessments
Schools Like Ours National Assessments
Purpose To hold schools accountable for
improving achievement in reading
and mathematics by comparing their
performance to other similar schools,
and setting targets in the context of a
three-year plan.
To monitor achievement in English
reading and mathematics over time
relative to targets specified on page 10 of
the Draft National Plan.
Domains assessed English reading and mathematics English reading and mathematics
Stakes High-stakes, with school-level
outcomes reported to Boards of
Management, Inspectorate
Low stakes, with system-level outcomes
reported
Frequency Annually Every four years
Grade levels involved 2nd, 4th, 6th Primary, 2nd Year post-
primary
2nd, 4th, 6th Primary, 2nd Year post-
primary
National reporting Report nationally-aggregated data on
performance standards
Report nationally-aggregated data on
performance standards.
Proposed CPD and
support
CPD opportunities and information
to teachers and Boards of
Management on the use of
aggregated data from standardised
tests to enable reflection on the
impact of their teaching and learning
None proposed
On page 39 of the Draft Plan, it is noted that „schools also have access to reports published by
the ERC on national assessments of reading and mathematics‟. Our view is that there would
be value in including detailed information on the types of texts and items used in national
assessments, and on the outcomes, as part of the CPD in literacy and numeracy for teachers.
This occurs in New Zealand, in the context of tests administered as part of the National
Monitoring Project. It ensures that teachers are familiar with the purpose and uses of the
national assessments and that they can use released materials to compare the performance of
their students against national performance in a low-stakes context (International Reading
Association, 2010).
International Assessments
In spring 2011, Ireland will participate in two international studies of educational
achievement – the Progress in International Literacy Study (PIRLS) and the Trends in
International Mathematics and Science Study (TIMSS). Rather than expanding the national
20
assessment programme to include Fourth class, there may be value in participating in PIRLS
and TIMSS on a more regular basis than has been the case in the past (these studies are
implemented by the International Association for the Evaluation of Educational Achievement
every four or five years, but the last time Ireland participated was in 1995). Regular
participation in PIRLS and TIMSS would provide valuable comparative information against
which to benchmark the performance of Fourth class students in Ireland, and to verify if any
increases in performance on tests of literacy and numeracy administered by schools are
transferred to international tests and vice versa8. It might also be pointed out that access to the
outcomes of international assessments such as PIRLS and TIMSS could inform curriculum
development and renewal as PIRLS and TIMSS compare the curricula of participating
countries in a systematic way (e.g., Kennedy, Mullis, Martin & Trong, 2007).
Reviews of the Irish outcomes of PISA 2009 revealed some issues relating to the procedures
used by the OECD and its contractors to scale PISA reading literacy across assessment cycles
(Cosgrove et al., 2010; LaRoche & Cartwright, 2010). It would seem important to see that
these technical issues are addressed in future PISA cycles, so that greater confidence can be
placed in trend results issued by the OECD. Commentary in the media and elsewhere on the
outcomes of PISA also supported the view that there may be a gap between what
assessments like PISA actually do, and what people think they can do.
There is some evidence that the outcomes of PISA have had an impact on curriculum
development at junior cycle in Ireland (see Conway & Sloane, 2005 in respect of
mathematics and DES, 2003 in respect of science). There is value in considering implications
of the PISA reading literacy framework in reviewing and revising the Junior Certificate
English syllabus, particularly the recent widening of the concept of reading literacy in PISA
to include both print-based and electronic reading (OECD, 2009).
8 An evaluation of the benefits of participating in PIRLS and TIMSS in Fourth class could be done after the
reporting stage of PIRLS and TIMSS 2011.
21
7. Conclusion
While we are supportive of many of the proposals for assessment in the Draft Plan, we have
concerns about the schedule for implementing new assessment procedures, and believe that
some of the proposals for assessment should be the subject of further study before
implementation across all schools.
In Table 7-1, we have outlined the time scale for the introduction of all the assessments
initiatives proposed in Chapter 6 of the Draft Plan. We are not sure that the system can
sustain such a large number of additional assessments at this time, and suggest a more
gradual approach to introducing these measures. We also have reservations about the
methodology proposed for Schools Like Ours, and believe that some analysis work, mainly
using existing data sources, should be done before implementing Schools Like Ours.
The Centre proposes a more gradual introduction of assessment initiatives, with time
available to study the impact of some measures in a subset of schools before extending them
to all class levels and all schools:
Expand the programme of national assessments of English reading and mathematics
for the 2013 cycle. Some development work will be required if the assessments are to
be extended to Fourth class (where a secure English reading test is not currently
available) and Second year in post-primary schools (though the Educational Research
Centre has already begun work on tests for Second years).
Although national assessments involving samples of schools and students are low
stakes, they have the potential to provide valuable information on the functioning of
the educational system and on how students interact with the reading and mathematics
tasks they are assigned. We would like to see more information (outcomes, examples
of tasks etc.) from the national assessments featuring in CPD for teachers.
The scope of national assessments at some class levels might be broadened to look at
non-test outcomes, such as the teaching of English and mathematics in classrooms.
For example, a video-study of a subset of participating schools could provide valuable
quantitative and qualitative information on teaching methods in classrooms that could
then be disseminated to all teachers.
As part of the national assessments, the introduction of computer-based testing for
students in Second-year post-primary should be considered, with an extension of this
to other class levels in later cycles.
A study should be conducted that examines in more detail some of the assumptions
underlying the Schools Like Ours initiative. Some of the data required for such a study
already exist (e.g., Junior Certificate examination results9 for the same schools for
consecutive years, data on the same DEIS schools for 2007 and 2010). Additional
9 It might be noted that Junior Certificate results have the potential to provide far more detailed information
on student performance than standardised test results. In the JC examinations, students are assessed over a far broader range of outcomes than on a typical standardised test.
22
data, if needed, could be obtained by administering the national assessments in
consecutive years in a subsample of primary schools.
Table 7.1 Indicative Timeline for Implementation of Assessment Proposals in Draft Plan*
Standards-based
Assessments
Administered by
Teachers
Schools Like Ours
Standardised
Testing
National
Assessments of
English reading and
mathematics
International
Assessments
2010-2011 PIRLS and TIMSS to
be implemented in
Fourth class in
primary schools
2011-2012 Implementation of
Testing in 2nd, 4th,
6th Classes Primary
and 2nd year Post-
primary.
PISA 2012 (Major
domain: Mathematics)
15-year olds Post
Primary
2012-2013 Implementation of
Testing in 2nd, 4th,
6th Classes Primary
and 2nd year Post-
primary.
Implementation of
National Assessments
in Second, Fourth and
Sixth classes
(Primary) and Second
Year (Post-primary)
(not yet confirmed)
2013-2014 Implementation of
English Standards-
based Assessments
(Primary) (Senior
Infants, 2nd, 4th, 6th
classes)
Implementation of
Testing in 2nd, 4th,
6th Classes Primary
and 2nd year Post-
primary. Collection of
aggregated data –
English (Primary)
2014-2015 Implementation of
English and
Mathematics
Standards-based
Assessments
(Primary) (Senior
Infants, 2nd, 4th, 6th
classes)
Implementation of
Testing in 2nd, 4th,
6th Classes Primary
and 2nd year Post-
primary. Collection of
aggregated data –
English & Maths
(Primary)
TIMSS in Fourth
class in primary
schools (Participation
not confirmed);
PISA 2015 Main
Domain: Science. 15-
year olds Post
Primary
2015-2016 Implementation of
English and
Mathematics
Standards-based
Assessments
(Primary) and English
(Post-primary)
Implementation of
Testing in 2nd, 4th,
6th Classes Primary
and 2nd year Post-
primary. Collection of
aggregated data –
English & Maths
(Primary); English
(Post-primary)
PIRLS in Fourth class
in primary schools
(Participation not
confirmed)
2016-2017 Implementation of
English and
Mathematics
Standards-based
Assessments (Primary
and Post-primary)
Implementation of
Testing in 2nd, 4th,
6th Classes Primary
and 2nd year Post-
primary. Collection of
aggregated data –
English & Maths
(Primary); English &
Maths (Post-primary)
Implementation of
National Assessments
in Second, Fourth and
Sixth classes
(Primary) and Second
Year (Post-primary)
(not yet confirmed)
*Excludes piloting, test development, field trialling etc.
23
Some of the issues that could be addressed are variation in performance in schools
(especially smaller schools) over a two or three year period; numbers of schools that
deviate significantly from the average performance of other similar schools; average
expected progress in different school types; and, where available, information on the
effects of standard setting on performance and the engagement of different
stakeholders (school management, teachers, students, parents and community) in
supporting improvement.
Using the same databases, value-added studies should be conducted to examine the
effects of adjusting average school achievement scores using a continuous measure of
school average socioeconomic status and other relevant contextual variables10
. Is
multi-level modelling involving both school and student variables better than single-
level modelling that involves adjusting school average scores using school-level
average socioeconomic status only? How much error is associated with year-on-year
measures of value added using different approaches? Are there any additional benefits
to be had by measuring value-added differences from one year to the next, compared
with a longer interval such as three years? Should school scores be averaged over a
number of years? How can the difficulties associated with small cohort sizes be dealt
with within a value-added framework (for example, can or should clusters of adjacent
small schools be collapsed into a single group)?
Examine how new measures of social context that can be updated regularly might be
developed. Summary school scores on the DEIS contextual variables were computed
in 2005 and are probably now out of date in respect of some schools. If schools are re-
evaluated for participation in DEIS or other such initiatives in the next few years, it
would be important to examine how a measure of school social context that can be
updated on a more regular basis could be constructed. A primary pupil database,
updated on an annual basis, would yield enhanced records of pupil characteristics that
could be anonymised and then used in analysing school-level performance. Other
methods, such as use of street addresses to provide an indicator of socio-economic
status, can be expected to emerge over time.11
Starting in 2011-12, design a programme for strengthening teachers‟ classroom-based
assessments. Begin implementation of the programme, perhaps as part of CPD in
literacy and numeracy, in 2012-13. While such a programme might initially focus on
general assessment skills (such as those outlined in Section 2 of this response, and
elaborated on in Assessment in the Primary School Curriculum – Guidelines for
Schools), it could be broadened over time to include outcomes of national assessments
as well as implementation and interpretation of classroom assessments based on
national standards.
10
The Centre has already done some relevant analyses. 11
The work of the group put in place by the Social Inclusion Unit of the DES is relevant here.
24
References
Afflerbach, P., & Cho, B. Y. (2011). The classroom assessment of reading. In M. L. Kamil, P. D.
Pearson, E.B. Moje & P. P. Afflerbach (Eds.), Handbook of reading research (Vol. 4), 487-
514. New York: Routledge.
Assessment Reform Group. (2006). The role of teachers in the assessment of learning. Cambridge:
Author.
Black P.J. & Wiliam. D. (1998), Assessment and classroom learning. Assessment in Education, 5(1),
7-74.
Braun, Chudowsky, & Koenig, (2010). Getting value out of value-added: Report of a workshop.
Washington, DC: National Academic Press.
Conway, P. F. and Sloane, F. C. (2006) International Trends in Post-Primary Mathematics
Education: Perspectives on Learning, Teaching and Assessment. Dublin: National Council for
Curriculum and Assessment.
Cosgrove, J., Shiel, G., Archer, P., & Perkins, R. (2010). Comparisons of performance in Ireland on
PISA 2000 and PISA 2009: A preliminary report to the Department of Educational and Skills.
Dublin: Educational Research Centre.
DES. (Department of Education and Science). (2005). An evaluation of curriculum implementation in
primary schools: English, mathematics and visual arts. Dublin: Government Publications.
DES. (2006). Circular 0138/2006. Supporting assessment in primary schools. Dublin: Author.
DES. (2003). Junior Certificate science syllabus. Ordinary level and higher level. Dublin: Stationery
Office.
DES. Inspectorate. (No date). Looking at English: Teaching and learning English at post-primary
level. Dublin: Author.
Educational Research Centre. (2008). Drumcondra primary mathematics test-revised. Progress in
mathematics: Using class-level performance bands and monitoring progress over time.
Dublin: Author. http://www.erc.ie/documents/dpmt_combined_documents.pdf
Educational Research Centre. (2010). Drumcondra tests of early literacy – screening and diagnostic.
Dublin: Author.
Educational Research Centre. (2011). Drumcondra tests of early numeracy – screening and
diagnostic. Dublin: Author.
Eivers, E., Shiel, G., & Shortt, F. (2004). Reading literacy in disadvantaged primary schools. Dublin:
Educational Research Centre.
Eivers, E., Shiel, G., Perkins, R., & Cosgrove, J. (2005). The 2004 National Assessment of English
Reading. Dublin: Educational Research Centre.
Eivers, E., Close, S., Shiel, G., Millar, D., Clerkin, A., Gilleece, L., & Kiniry, J. (2010). The 2009
National Assessments of mathematics and English reading. Dublin: Stationery Office.
Accessed at: http://www.erc.ie/documents/na2009_report.pdf
25
Gleeson, J., Kelly, J., & Archer, P. (1988a). A structured program for infant classrooms in
mathematics, reading and oral language. Volume 1 – Manual for junior infants. Dublin: The
Department of Education in association with the Educational Research Centre.
Gleeson, J., Kelly, J., & Archer, P. (1988b). A structured program for infant classrooms in
mathematics, reading and oral language. Volume 2 – Manual for senior infants. Dublin: The
Department of Education in association with the Educational Research Centre.
Gleeson, J., Kelly, J., & Archer, P. (1988c). A structured program for infant classrooms in
mathematics, reading and oral language. Volume 3 – Manual for first class. Dublin: The
Department of Education in association with the Educational Research Centre.
House of Commons. (2008). Third Report of the Select Committee on Children, Schools and Families.
UK Parliament, May.
International Reading Association. (2010). Standards for the assessment of reading and writing.
Newark, DE: Author.
http://www.reading.org/general/currentresearch/standards/assessmentstandards.aspx
Kellaghan, T., Madaus, G.F., & Airasian, P.W. (1980). Standardised testingin elementary schools:
Effects on schools, teachers, and students. Washington DC: National Institute of Education,
US Department of Health, Education & Welfare.
Kennedy, A.M., Mullis, I.V.S., Martin, M.O. & Trong, K.I. (2007). PIRLS 2006 Encyclopaedia: A
guide to reading education in 40 PIRLS 2006 countries. Chestnut Hill, MA: TIMSS and
PIRLS International Study Centre.
LaRoche, S., & Cartwright, F. (2010). Independent review of the 2009 PISA results for Ireland:
Report prepared for the Educational Research Centre at the request of the Department of
Education and Skills. Dublin: Department of Education and Skills.
National Council for Curriculum and Assessment (NCCA). (2005). Primary curriculum review.
Phase 1. Report. Dublin: Author.
National Council for Curriculum and Assessment. (2007). Assessment in the primary school
curriculum – Guidelines for schools. Dublin: Author.
National Council for Curriculum and Assessment (NCCA). (2008). Primary curriculum review.
Phase 2. Report with recommendations. Dublin: Author.
Ó Siaghail, G., & Deiseach, C. (2004). Próifílí measúnachta don Ghaeilge sna scoileanna Gaeltachta
agus scoileanna lán-Ghaeilge. Lámhleabhar. Baile Átha Cliath: Foras Taighde ar Oideachas.
OECD. (Organisation for Economic Cooperation and Development). (2008). Measuring
improvements in learning outcomes: Best practices to assess the value-added of schools.
Paris: Author.
OECD. (2009). PISA 2009 Assessment framework: Key competencies in reading, mathematics and
science. Paris: Author.
Perkins, R., Moran, G., Cosgrove, J., & Shiel, G. (2010). PISA 2009: The performance and progress
of Irish 15-year olds in Ireland. Dublin: Educational Research Centre.
Queensland Studies Authority. (2009). Student assessment regimes: Getting the balance right for
Australia.
http://www.qsa.qld.edu.au/downloads/publications/qsa_paper_assess_balance_aust.pdf
26
Ray, A. (2006). School Value Added Measures in England: A Paper for the OECD Project on the
Development of Value-Added Models in Education Systems. London: Department for
education and Skills.
Shiel, G., & Murphy, R. (2000). Drumcondra English profiles. A framework for assessing oral
language, reading and writing in primary schools. Dublin: Educational Research Centre.
Shiel, G., Surgenor, P., Close, S., & Millar, D. (2006). The 2004 National Assessment of Mathematics
Achievement. Dublin: Educational Research Centre.
Shiel, G., Kellaghan, T., & Moran, G. (2010). Standardised testing in lower secondary education
(Research report no. 12). Dublin: National Council for Curriculum and Assessment.
http://www.ncca.ie/en/Publications/Reports/Standardised_Testing_In_LowerSecondary_Educ
ation.pdf
State Examinations Commission. (2010a). State examination statistics – Provisional results for Junior
Certificate 2010. http://www.examinations.ie/statistics/statistics_2010/JC_619_2010.pdf
State Examinations Commission. (2010b). 2010 Leaving Certificate statistics.
http://www.examinations.ie/statistics/statistics_2010/gender_ard_2010_excluding_less_than_
10.pdf
http://www.examinations.ie/statistics/statistics_2010/gender_gnath_2010_excluding_less_tha
n_10.pdf
http://www.examinations.ie/statistics/statistics_2010/gender_bonn_2010_excluding_less_than
_10.pdf
Thomas, SM, Peng, W-J & Gray, J. (2007). Modelling patterns of improvement over time: value
added trends in English secondary school performance across ten cohorts. Oxford Review of
Education, 33 (3), 261-295.
Wandall, J. (2008). National tests in Denmark. CAT as a pedagogical tool. In F. Scheuermann & J.
Björnsson (Eds.), The transition to computer-based assessment: New approaches to skills
assessment and implications for large-scale assessment. ISPRA, Italy: European Commission
Joint Research Centre/Institute for the Protection and Security of the Citizen.
