1

National Standardisation Group Meeting 6

Thursday 6th August

2.00 – 4.40 p.m. via TEAMS.

Attendees:

Andrea Feeney Director CGEO, Chairperson

Elaine Sheridan Asst. Director CGEO

Hugh McManus Asst. Director CGEO

Dr David Millar Assessment Manager, CGEO

Aidan Farrell CEO, State Examinations Commission

Dr Jude Cosgrove Director, Educational Research Centre

Dr Harold Hislop Chief Inspector, Inspectorate, Dept of Education

Orlaith O’Connor Asst Chief Inspector, Inspectorate, Dept of Education

Fernando Cartwright President and CEO Polymetrika International Incorporated.

Dr Kentaro Yamamoto, Technical Advisor.

Paddy Quinn Asst Principal, CGEO, Secretary to the Group

1. At the outset the Chair informed the meeting that the Calculated Grades Programme

Board had been notified on 30th July of the NSG’s change to way of working and

accepted the rationale behind the change.

2. It was also noted that the Scottish Qualification Authority had released their A Level and

GCSE. There had been a high degree of media attention especially on the adjustments to

teacher estimates focused on socio-economic status. . The CGEO will be watching

developments in Scotland and also England, Wales and Norther Ireland when their

results are released on 13/8.

3. A results test file has now been returned form PII for testing on the CGEO database.

4. With the ongoing legal case, any forthcoming announcements re calculated grades will

require careful thought.

5. A number of draft documents had been circulated in advance of the meeting, The Chair

noted with thanks the contribution of CGEO colleagues in the development of these

documents.

Minutes from meeting 5 and matters arising

6. The draft minutes of meeting 5 were agreed and adopted. All matters arising form part

of the agenda.

Schools v Class level Conditioning Distributions

7. Following the last meeting, CGEO had prepared a short document setting out the

approach to conditioning at school v class level, some detail of which had been provided

in the Decision-Making Framework document. This document had been circulated very

2

shortly before the meeting and was not considered in detail. Group members were

invited to provide any feedback on the document via e-mail after the meeting. The

Chairperson noted that while the approach was for approval the overall approach to

standardisation at school level was implicit both in the model design and in the

information published about calculated grades.

Review of Data Models - Overview of outcomes from Pass Through and Estimation Models

to date

8. It was noted that, while there are models up to and including 13 (d), the focus of the

NSG should remain on version 7 as the subsequent versions are test versions. The group

will be updated when later models are available for analysis.

9. An overview was provided of model development to date and progress towards

achieving the objectives. This was a recap of the issues considered by the group already

to date, and extended onwards to the features that were evident in model 7.

10. By way of clarification with regard to the question of whether different models might

conceivably apply across different subjects and/or levels, it was noted and agreed that

the objective is to develop one high quality, generic model with global application,

which blends the four model components to best effect. The application of the model is

context-specific and the particular conditioning data that arises in different schools and

different subjects will give rise to school-level and cross-subject differences as regards

how the different components influence the calculated scores at the different points in

the mark distribution. The difficulty of communicating the model – particularly given

the level of sophistication required in the model in order to maximise fairness in the

standardisation process – was acknowledged.

11. There was some discussion of the degree to which ‘cohort shift’ is taken account of in

the model and/or will need to be addressed more explicitly later, given the lower

propensity of students to drop levels this year, leading to a higher proportion of

students at higher level and lower proportions at Ordinary level and Foundation levels.

This is more evident in some subjects than others. The cohort change arising from this

behavioural change on the part of students is greatest in the Foundation levels in Irish

and Maths, in which the normal candidature has been halved.

12. The NSG noted the focus that there has been on differential effects by demographic

characteristics following the release of the results in Scotland, particularly socio-

economic characteristics. It was noted that examination of such differential effects is

already part of the model validation process.

Review of Data Models – Overview by Polymetrika

13. Overview of strategy

To combine

National distribution National performance by subject from previous years

Use inverse variance weighting to combine these elements in the School-level prior attainment

distribution Prediction of LC results by subject based on prior JC performance of the

3

students in the group aggregated to the school level

“most trustworthy” way.

School historical distribution LC performance in the school from previous years

Teacher estimates Including both rank order and interval-level information contained within the estimated percentage marks

14. Model 5 v Model 7

The national and school regression based estimates were weighted too highly in the early

models. Model 7 changed the weight of how the elements fed in.

15. Interim Model 13 variants

The credibility of the national results can be most readily observed in the scatterplots of

school averages in a “normal” year with the school averages from the model. The goal is to

achieve a similar pattern of results without forcing the model distributions to match

historical patterns. Regression toward the mean will always cause some mismatch – with

the scatterplots not falling exactly on the 45° diagonal. It will be difficult to reconcile the

bottom tail of the distribution as there are few candidates there in the teacher estimates.

There will likely be heavy tails at the top and bottom of the distribution even with the best

model arising from the strategy in play for the current phase of development.

Dr Yamamoto has suggested explicitly building in variance decomposition into the model but

this has not yet been done.

16. Coming up

Polymetrika has generalised the app to allow for the easier implementation of modelling of

the LCA results.

Work has begun on the composite measure for Junior Certificate prior attainment, intended

to improve the quality of the regression-based predictor.

Polymetrika is working on further models 14, 15 and 16 (working labels which may change).

Model 14 involved a restructuring to optimise the running of the algorithm. Model 15 will

test the balance of the variance components and will not be run on many subjects. Model

16 will include the Junior Certificate composite in the predictor and is likely to be the next

milepost for broad checking of calculated marks. It is also anticipated that Model 16 will

allow the running of LCA estimations, although it was also noted the most efficient strategy

in the long run is to have the model fully or almost fully refined before running it on LCA

data.

17. Points noted during discussion

It was noted that one question that would continue to be asked by stakeholders is: what is

the relative weighting of the different elements that feed into the calculated marks? The

difficulty with answering this question in light of the fact that the weighting varies according

to the relative quality of the information from each source at each point in the distribution

4

for each subject and level was acknowledged. Nevertheless, although the contribution of

the four elements will vary in these ways, it was also noted that it will be possible to report

national averages for the relative contributions.

It was again noted that poor performance in the Junior Certificate exams does not

“condemn” a candidate under these models to a poor calculated mark. Junior Certificate

information is fed into the model at a school level rather than an individual one, and that

they are furthermore always based on a distribution and not a point estimate. If there is

little school information, population information is given greater weight. If there is plenty of

information on the school distribution (stability), the population information is given less

weight.

It was noted that the model is run for each subject separately. That is, the ‘school historical’

distribution involved is the one for the subject and level concerned, not one that aggregates

historical performance across subjects.

It was also noted that the better use of prior attainment data may help deal with the much

reduced numbers at Foundation level and the non-random missingness in the NCLs, but this

will need to be kept under review.

Treatment of Small Classes and Subjects Studied Outside of School

The CGEO paper, Treatment of Students in Small Classes and of Subjects Studied Outside of

School (including Out-of-School Learners) in the National Standardisation Process, was

discussed.

18. This treatment of these cohorts within the statistical model is a matter for decision of

the NSG. At the outset, the technical and detailed nature of the documents circulated

in advance of the meeting were acknowledged. The degree of technicality is needed to

give the group full visibility of the proposal. In discussion of the document the following

issues were noted.

The paper was considered very helpful but the addition of the data about the

number of students affected by the decision would assist in understanding

the context

The method and rationale for imputing data from elsewhere for small class

groups was discussed

The rationale for allowing the use of individual prior attainment data and

concurrent other-subject data in the estimation model – both of which are

excluded from use in the generality of cases – for students studying a subject

which is not taught in (the school outside of school) and for out-of-school

learners was discussed.

It was generally acknowledged that it would be unfair to the generality of

students to have any particular cohort not subject to some form of

standardisation. Without some intervention, there is a risk that the

teacher/tutor estimate will become the final mark in cases where this is not

appropriate for these groups of students.

5

Based on the discussion at the meeting the NSG approved the approach, as documented in

the draft paper, to proceed. An updated version of the document will be circulated that will

take account of some observations made at the meeting.

Treatment of Non-Curricular Language Subjects

The CGEO paper National Standardisation and the Non-Curricular EU Languages was

discussed.

19. The following points were noted:

A credible methodology is required for dealing with NCL’s.

The impact of the cohort change needs to be taken account of in the model in a way

which is fair to these students.

That there are some subjects with larger cohorts which could be treated as individual

subjects was noted.

The arguments presented in the paper are very persuasive and strong and the logic

for treating all NCLs as a single subject for conditioning was accepted on the grounds

of fairness.

It was noted that students studying these subjects will, in light of the decision re

subjects taken outside of school, have their individual prior attainment data and

concurrent other-subject data taken into account in the estimation model.

The potential need for further intervention in respect of the macro-level distribution as set

out in the paper was noted. If it is necessary to use soft evidence to change the distribution

(on the grounds that the application of the normal national distribution may disadvantage

this year’s cohort) then this will need to documented at that stage in the process if this

becomes necessary.

20. The Chair drew the attention of the Group to the approach noting the potential for a

similar intervention to be necessary for Foundation Level Maths and Irish given the

degree of cohort shift experienced in those subjects.

Junior Cycle Composite Measure

Following on from the last meeting, the CGEO paper Junior Certificate/Cycle composite input

measure was discussed. This is a matter for noting.

21. The paper describes the creation of a common scale and subsequent composite measure to

improve the predictive power of prior performance at Junior Cycle

22. This has been developed in order to address a technical problem that arises from the

sparseness of the data (and the particular patterns of sparseness) in the student–

subject matrix of JC prior attainment data.

23. The composite approach identified should improve the accuracy of the model and hence the

accuracy of the calculated marks and, ultimately, calculated grades. This is in the interest of

fairness and accuracy of the model.

6

24. The common scale is based on grade threshold alignments implied by the ERC’s JOPS

score which has been used for various studies in the past and which has reputational

value.

25. The proposed approach is to use performance in Irish, English, Maths and the

student’s two best other subjects, excluding CSPE.

Update on Preliminary Report to the Independent Steering Committee.

26. The interim report has been presented to the ISC. While no formal response has been

received, Dr Áine Lawlor, Chair of the ISC, complimented the group on their work to date

and wished them well going forward.

Update on Leaving Certificate Applied.

27. The LCA file has transferred to Polymetrika but as the Leaving Certificate is being

prioritised at the moment, it has not yet been put through any modelling.

28. The LCA should provide a solid basis for the application of the model due to having more

data on which to base the modelling.

29. The work of the CGEO team in cleaning up the LCA data was acknowledged.

30. The hope is that there will be a model to review in the coming week. However, the

greatest efficiency in progressing the overall work in the long run will arise from getting

the model to as final a state as possible before running it on LCA data.

31. The Chair acknowledged that LCA is extremely complex in its make-up and complimented

all on the work to date.

AOB

32. Nothing arose under AOB

In closing the meeting, the Chair outlined that as the group progresses in the standardisation

phase towards sign-off that the expectation is that the meetings would be longer.

The next meeting will be held on Thursday 13th August.

Analysis should continue on version 7 of the model with members to be notified if a later

version becomes available.

The meeting concluded at 4.40 p.m.

Decisions

The Group agreed that, while the latest run of the model had made good progress in

aligning outcomes at a macro level, further iterations of the model are required.

7

School V Class Level Conditioning Distribution document to be reviewed by members with

comments returned by e-mail.

Documents on treatment of

a) Small Classes and Subjects Outside of School and

b) NCL’s

agreed in principle but to be updated to reflect discussion.

Actions

1. Comments to be returned on note describing the process of conditioning at school

level and retaining rank order.

2. Note on an approach to an aggregate for Junior Cycle to be updated.

3. Note on the treatment of small class sizes including class sizes of one to be updated.

4. Note on the treatment of the NCL’s to be updated.

5. Group to continue to analyse version 7 of the model with notification to issue when

an updated model becomes available.