Understanding the Work of the HKEAA:Mark Processing and Grading

Mark processingSome important steps after completion of marking:Equating of marks of optional parts / papersModeration of SBA marks submitted by schools

Graded Approach for English Language Papers 1 & 3

Why do we need a graded approach?Adopted in Paper 1 (Reading) and Paper 3 (Listening and Integrated Skills)Compulsory Part A and either Part B1 (easier) or Part B2 (more difficult)Cater to candidates with a wide range of ability

What are the advantages?No labeling effect No need to extend examination time or reduce coverage of the curriculumCandidates need not indicate choice at registration so possible to catch up before the examinationStatistical equating to ensure that candidates receive comparable scores on the same scale whether they opt for the easier or more difficult part

Why is there a capping policy?Candidates attempting Parts A and B2 will be able to attain the highest level of performance (Levels 5 and 5* / 5**) Candidates who attempt Parts A and B1 will only be able to attain up to Level 4 in the relevant component Reason for capping policy is that items in B1 do not assess skills covered by Level 5 descriptors

Why is equating necessary?Easier to get high marks in Part B1 than in Part B2 because the items are easier1 mark in B1 is not equal to 1 mark in B2Need to convert the marks to the same scaleUse the marks of Part A as the mediator because all candidates do Part A

Example100 candidates got 36 marks in Part A50 of them did B1 and 50 did B2The 50 B1 candidates got 40 marks in B1The 50 B2 candidates got 32 marks in B2 40 marks in B1 = 32 marks in B2

What is equipercentile equating?Large candidature for English Language (e.g. ~ 80,000 in 2013)All candidates do Part A and can be rank ordered based on their marks in Part AB1 marks are statistically adjusted for difficulty to the same level as B2 marks using the percentile ranks of candidates performance in Part A and Part B1/B2

Equipercentile method of equatingB1 candidates ranking in Part B1

50th%

B1 candidates ranking in Part A

50th%

B2 candidates ranking in Part A

40th%

B2 candidates ranking in Part B2

40th%

40363632

Equipercentile equatinghttp://www.hkeaa.edu.hk/DocLibrary/Media/Leaflets/HKDSE_SRR_A4_Booklet_Jun2011.pdf

Is there a fixed conversion factor?The examination papers for each year will be different from the previous yearsTherefore no fixed conversion factorA conversion table will be produced every year for that particular examination paper based on the performance of the candidates in Parts A, B1 and B2

How to choose between B1 and B2?Levels 3-4 Levels 1-3 Levels 4-5Parts A & B2 (more difficult part) Parts A and B1 (easier part) Parts A & B1 or Parts A and B2

How is the paper mark calculated?For Part B1 candidates:Part A marks + Part B2 equivalent marksFor Part B2 candidates:Part A marks + Part B2 marksTherefore, choosing B1 or B2 makes no difference for mid-range candidates

Moderation of SBA Marks

Some queries/concernsWhy moderate schools SBA marks?Is it fair to moderate students SBA marks based on exam results?

Why moderation?Teachers know their students well and thus are best placed to judge their relative performance within a schoolHowever, they are not necessarily aware of the standards of performance across all schoolsModeration ensures the consistency of assessment standards across schools in order to achieve across-school comparability

What are the principles of SBA moderation?Accept teacher judgments of relative student performance within a school (internal standardisation, student queries)Moderation is done on school basis (not on individual basis) to standardise across schools

Methods of SBA moderationStatisticalThe distribution of a schools SBA marks for a subject is re-shaped taking into consideration schools pattern of mark distribution for that subjects examination papersExpert judgmentA team of experienced, subject experts reviews samples of schools SBA work for a subject

Moderation methods

Features of statistical moderationInternal ranking would not be changedThe statistical moderation method would consider the empirical correlation between examination results and SBA resultsEliminate the effect of strictness and leniency

Features of statistical moderationSample review to help identify outlier schoolsFor outlier schools, adjustment will be carried out, if necessary, by considering a number of factors, such as moderated marks based on examination results and sample review findings so as to reflect the actual SBA performance

Expert judgment moderation Review of samples of students workDetermination of school performance level on SBA and individual students moderated scoresDetection of cases with extreme adjustment and follow-up actions

Review of samples of students workStratified random sampling within each schoolDistrict Coordinators / External Assessors review samples with reference to the stipulated assessment criteria to determine the group performance levelMore samples from a school may be collected if necessary

Result of moderation in 2012 HKDSEasexpectedMean SBA Adjustment

Result of moderation in 2012 HKDSEasexpectedMean SBA Adjustment

Result of moderation in 2012 HKDSEIn general, most teachers are aware of the standards and their SBA marks need very little adjustmentSmall deviation from the standard is expected due to normal variation of experience and subjective judgment: no systemic bias observed Moderation provides assurance of fairness to students, parents, and schools

ReferenceHKDSE Examination: Moderation of School-based Assessment Scores http://www.hkeaa.edu.hk/DocLibrary/SBA/HKDSE/HKDSE-SBA-ModerationBooklet_r.pdf

Standards-referenced Reporting

Some queries/concernsWhy the grade awards are different across subjects?

Is this unfair to my subject?

SubjectNo. SatChinese Version% Percentage of levels awarded5**5*+5+4+3+2+1+U

Information and Communication Technology7 91861.60.72.97.225.250.977.593.07.0 Liberal Studies69 72188.00.83.07.735.772.590.898.91.1 Literature in English 451 -2.710.425.957.680.592.596.93.1Mathematics Compulsory Part69 29949.51.14.812.134.157.179.791.78.3Extended Part (Calculus and Statistics)7 34727.11.45.915.143.469.786.995.14.9Extended Part (Algebra and Calculus)8 09118.13.012.031.456.179.892.598.02.0 Music 232 8.61.77.819.060.885.897.899.60.4 Physical Education 68193.50.10.41.69.337.763.988.711.3 Physics15 23331.42.49.623.848.873.088.897.03.0

*Standards-referenced reporting system 24 NSS SubjectsDescriptors and exemplars to indicate standards required for each levelFacilitate users, including tertiary institutions and employers, to set appropriate entrance/ job requirements

Standards-referenced reporting system 24 NSS SubjectsAdopted to report candidates' assessment results in all 24 HKDSE Category A subjects

SRR adopted for HKCE English Language and Chinese Language since 2007

Grading in the HKDSECut scores and grade levels determined so that each level represents a fixed standard of achievement from year to yearMethodology combines psychometric measurement and expert judgmentPsychometric measurements:i)Monitoring tests for the core subjectsii)Group ability index (GAI) for elective subjects

Grading in the HKDSEExpert judgments made independently by experienced examiners and experts based on the actual performance of candidates on the exams, using the level descriptors and library scripts for referencePsychometric cut score recommendations are derived for the reference of the expert panelNo pre-set quota for each level /subject

Standards maintenance in 2013Four core subjects (CEML)Additional reference made to:statistical data from the monitoring tests which were administered to a representative sample of 2012 and 2013 DSE school candidates

Using monitoring tests to maintain standards Statistically-generated cut scores for reference of Expert Panel

Different Category A elective subjects may attract students of different abilitiesCompute an indicator (index) to reflect overall performance in core subjects (ability) of all candidates taking the subject (group), taking into account how the performance in the elective correlates with each of the core subjectsProvided as supplementary statistics, in addition to the level descriptors, library scripts and live scripts, to the subject experts for their expert judgment. Final decision based on performance as reflected in the candidates scripts

Group Ability Index (GAI) for elective subjects

Group Ability IndexThe GAI of Subject X for a certain level (e.g. Level 3) or above is defined as:n/NProportion of candidates in a group attained a particular level (e.g. Level 3) for a core subjectbCoefficients obtained by regressing scores of Subject X on scores of CEML

Grading Methodology and Procedures

Who are involved in gradingEach subject sets up its expert panel which comprises:the chief examiner(s) of different papers, assistant examiners/markers, if necessary;SBA supervisors/chief project examiner, if applicable, andsubject manager

Overview of grading proceduresFamiliarisation with standardsSample script selectionStandardisationPost-marking exerciseExpert Panel MeetingInternal MeetingPublic Examinations Board Meeting

4. Post-marking exerciseDetermine preliminary cut scores ranges for all levels for each exam paper (incl. SBA) with reference to:the difficulty of live question paper and requirements of marking scheme inter-paper correlations, mean and standard deviation of current years papers in comparison to previous years item statistics, if available impression of candidates performance when marking live scripts or reviewing students SBA work level descriptors and borderline performance samples in the previous exams

5. Expert panel meetingObjective:Apply expert judgment to arrive at a set of recommended cut scores for different levels for the subject and its components, if applicable Make reference to marked live scripts falling within / around the recommended cut score ranges and exam statistics

Expert panel meeting (contd)Also refer to cut scores suggested by statistical means:for core subjects, the suggested cut scores are derived from the results of the monitoring tests (2013 data in comparison with the 2012 data) for elective subjects, the suggested cut scores are derived from GAI

SubjectNo. SatChinese Version% Percentage of levels awarded5**5*+5+4+3+2+1+U

Information and Communication Technology7 91861.60.72.97.225.250.977.593.07.0 Liberal Studies69 72188.00.83.07.735.772.590.898.91.1 Literature in English 451 -2.710.425.957.680.592.596.93.1Mathematics Compulsory Part69 29949.51.14.812.134.157.179.791.78.3Extended Part (Calculus and Statistics)7 34727.11.45.915.143.469.786.995.14.9Extended Part (Algebra and Calculus)8 09118.13.012.031.456.179.892.598.02.0 Music 232 8.61.77.819.060.885.897.899.60.4 Physical Education 68193.50.10.41.69.337.763.988.711.3 Physics15 23331.42.49.623.848.873.088.897.03.0

ReferenceGrading Procedures and Standards-reference Reporting in the HKDSE Examination http://www.hkeaa.edu.hk/DocLibrary/Media/Leaflets/HKDSE_SRR_A4_Booklet_Jun2011.pdf

Reporting of Results

Reporting of HKDSE resultsFor Category A subjects, results reported mostly on subject basis Component results reported for:Chinese LanguageEnglish LanguageCombined Science

Reasons for component reportingAllow end users to understand the achievement of candidates in different areas of the subject e.g. reading, writing, listening and speaking skills in language subjects different half subjects in Combined ScienceFor selection for studies or employment

FAQ on component result and subject resultWhy is my subject level lower than the average of individual component levels?

The factsSubject level is based on the overall subject mark which is the combination of all component marksNOT a simple averaging of component levels achieved

The factsEach level covers a range of marks; a candidate may have a score close to the lower/higher end of the rangeComponents of a subject carry different weightings; the marks for each component go through a weighting procedure before combining them to arrive at the subject mark

Example: HKDSE English LanguageComprises four components:Component 1: Reading (20%)Component 2: Writing (25%)Component 3: Listening and Integrated Skills (30%)Component 4: Speaking + SBA (25%)

*****

5

4321

Level achievedReading(20%)Writing(25%)Listening(30%)Speaking(25%)SubjectStudent A*55454Student B53545

Thank You

*********