Race to the TopAssessment

ProgramConsultation

Carol Campbell

Atlanta, GeorgiaNovember 17, 2009

2

Assessment Design Principles

EffectiveAssessment

Systems

Designed around a clearvision of purposeand learning goals

Align with and advance commonstandards, curriculumexpectations, learningobjectives and instructional strategies

Encompass and combineassessment of, for and as learning

Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning

Embody high expectations to engage, motivate, support and stretch all students to progress and achieve

Build and developprofessional capacity

Assess a range of content, knowledge, skills and performancein authentic, applied and appropriate ways

Are part of a clear and flexible learning trajectory across subjects and over education/career pathways

Support data-informed decision-making for local and system improvement

Place qualityof assessment overquantity

3

1. Clear Vision of Purpose and Learning Goals - Specifying 21st century skills, college and career-readiness

Example: High school examinations in England combine:Subject/curricular knowledge with generic skills Functional skills – “essential for life, learning and work” – practical

skills in English, mathematics and information and communication technology” (ICT). Example – Math level 1 - “A member of a reprographics team is able to look at statistics and graphs detailing paper usage and identify trends that might suggest that equipment requires servicing”

Personal learning and thinking skills (PLTS) – “These are skills that will equip people for successful employment and lifelong learning…PLTS require learners to be:

Independent enquirers Creative thinkers Reflective learners Team workers Self-managers Effective participants”

4

Assessing 21st Century Skills – England, Ages 14 - 19

Assessment criteria for Functional Skills – “Assessment must, according the skill areas:

Provide realistic contexts, scenarios and problems Specify tasks that are relevant to the context Require application of knowledge, skills and understanding for a

purpose Require problem solving Assess process skills and the outcome of their application in

different contextsGenerally assessments externally defined (exam boards) and

internally assessed (teachers), including controlled assessments in classroom and projects. Combinations of open and fixed response items varies by subject area and level.

Performance assessments increasingly used in range of qualifications – coursework, projects, extended writing

Opportunities for accredited work experience in some qualifications, e.g. Diploma composite qualification includes 10 days work experience

5

Ontario, Canada - Achievement Chart – English, Grades 9-12 – Thinking SkillsThinking – The use of critical and creative thinking skills and/or processes

Categories 50-59% (L1) 60-69% (L 2) 70-79% (L 3) 80-100% (L4)

The student:

Use of planning skills (e.g., generating ideas, gathering information, focusing research, organizing information)

uses planning skills with limited effectiveness

… some effectiveness

… considerable effectiveness

… high degrees of effectiveness

Use of processing skills (e.g. drawing inferences, interpreting, analysing, synthesizing, evaluating)

uses processing skills with limited effectiveness

… some effectiveness

… considerable effectiveness

… high degrees of effectiveness

Use of critical/creative thinking processes (e.g. oral discourse, research, critical analysis, critical literacy, metacognition, creative process)

uses critical/creative thinking processes with limited effectiveness

… some effectiveness

… considerable effectiveness

… high degrees of effectiveness

6

2. Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies

Example: Ontario, Canada Content Standards

Curriculum for Grades 1-12 for all subjects and courses Developed through extensive research, benchmarking,

and consultation with all stakeholders All teachers are required to teach the curriculum

expectations Performance Standards (criterion-referenced)

Criteria are described for student achievement of the curriculum expectations for Grades 1-12

Four levels of achievement Four categories of knowledge and skills: Knowledge and

Understanding; Thinking; Communication; Application Consistency:

Exemplars of student work for all curriculum Standard provincial report cards - aligned with content and

performance standards

7

3. Encompass and combine Assessment of, for and as learning

Assessment…Of learning - summative assessments to confirm

student knowledge, skills, understanding of curriculum, proficiency

For learning – formative assessments using information throughout the learning process to investigate student learning, inform instruction and provide feedback to support student progress

As learning – assessment as a process of meta-cognition involving student self-assessment and understanding of own learning processes and knowledge

8

Summative and Formative Assessments: Example – The National Assessment System, Scotland, UK

FORMATIVE

SUMMATIVE

INTERNAL EXTERNAL

Formative assessment

Personal learning planning

Involving learners, and parentsAnd other adults in the learningprocess

Local authority collection and analysis of information to inform provision and improvement

Inspection feedback and subject/quality/improving reports

Follow-through inspection activities

Teachers’ judgements and reports,With local moderation and NationalAssessments as part of understandingAnd sharing standards

Scottish Survey of Achievement P3, P5, P7, S2National Qualifications (SQA)International studiesHMIE inspections and reports on Authorities and schools

9

4. Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning

Multi-faceted assessments used internationally include:

Assessments of knowledge (recall & analysis) and assessments of performance (demonstration of ability to apply knowledge in practice)

Multiple-choice, constructed-response, extended tasks and projects

On-demand and curriculum-embedded elements Externally-developed (by teachers and

developers) and classroom-developed/managed Scoring made consistent through training,

moderation, calibration, and auditing

10

Examples of Assessment Types: Ontario and England

Types of Task Example

External standardized test – multiple choice response

Approx. 50% of questions in Grade 9 and 10 provincial assessments, OntarioNot used in England

External standardized test – open-ended response

Approx. 50% of questions in Grade 9 and 10 provincial assessments, OntarioUsed for KS 2, parts of GCSE (40-100% of final mark) and A-level in England

Performance assessment, e.g. project Controlled assessments (coursework) for GCSE (up to 60% of overall mark) and Extended Project for A-level (equivalent to half an A-level)

Interaction and observations between teacher and student, e.g. discussion, conference, classroom and homework tasks and tests

Systematic Teacher assessments reported to parents/guardians annually and data collected nationally at key points (England, Ontario)

Cumulative assessments, e.g. portfolios National Vocational Qualifications and Diplomas (England)

11

Singapore GCE A-Level Examinations

12

Systematic Achievement Reports at the End of Key Stages, England, UK

Key Stage 1 (age 6-7): Classroom evidence is combined with centrally-developed open-ended tests and tasks in English and mathematics, taken when children are ready and marked by teachers, with moderation.

Key Stage 2 (age 8-11): Teachers’ summary judgments of student progress from classroom evidence are combined with open-ended tests in English, mathematics and science; externally marked and results collected nationally.

Key Stage 3 (age 14): External tests were recently abolished. Moderated teacher assessments are used to report achievement levels in all subjects, nationally collected.

No mandatory high school assessments – although most students will participate in GCSEs (age 15-16) and progress to other academic (A/AS level), vocational (diploma) and flexible pathways for progression

13

5. Embody high expectations to engage, motivate, support and stretch all students to progress and achieve

Example: England, UK: Making Good Progress Pilot:

Progression Targets to increase the number of pupils who make two levels of progress throughout a key stage

Progression Premium to reward schools which help students who entered a key stage behind national expectations and make good progress

14

The Assessment for Learning Strategy, England, UKOur aims are that: Every child knows how they are doing, and understands

what they need to do to improve and how to get there. They get support they need to be motivated, independent learners on a ambitious trajectory of improvement

Every teacher is equipped to make well-founded judgement about pupils’ attainment, understands the concepts and principles of progression, and knows how to use their assessment judgements to forward plan, particularly for pupils who are not fulfilling their potential

Every school has in place structured and systematic assessment systems for making regular, useful, manageable and accurate assessments of pupils, and for tracking their progress

Every parent and carer knows how their child is doing, what they need to improve, and how they can support the child and their teachers.

15

Applying Knowledge:A Literacy Task, Grade 10 Ontario

Task: Write a minimum of three paragraphs expressing an opinion on the topic below. Develop your main idea with supporting details (proof, facts, examples, etc.).

Purpose and Audience: an adult who is interested in your opinion

Length: The lined space provided for your written work indicates the approximate length of the writing expected.

Topic: Are today’s famous people good role models for young people?

16

6. Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways

Example: General Certificate of Secondary Education (GCSE) (age 15-16), England

GCSEs are available in over 50 subjects: students choose. GCSEs normally take 2 years to complete. Many courses offer

assessment in each unit rather than only at the end. GCSEs use constructed response items, and usually include

structured classroom-based tasks that are more extended. Increased emphasis on extended writing and course work. Controlled assessments are taken under supervised conditions

and are either set by the awarding body and marked by teachers or set by teachers and marked by the awarding body. Teachers determine the timing of controlled assessments. Controlled assessments comprise up to 60% of a grade, depending on the subject.

Introduction of assessment of functional skills and personal learning and thinking skills across subjects

Cost of assessing one unit = $10 approx. Average costs of a GCSE course assessment = $40 approx.

17

ICT Controlled Assessment Task, GCSE EnglandLitchfield Promotions works with over 40 bands and artists to promote their

music and put on performances in the UK. The number of bands they have on their books is gradually expanding. Litchfield Promotions needs to be sure that each performance will make enough money to cover all the staffing costs and overheads as well as make a profit. Many people need to be paid: the bands; sound engineers; and, lighting technicians. There is also the cost of hiring the venue. Litchfield Promotions needs to create an ICT solution to ensure that they have all necessary information and that it is kept up to date. Their solution will show income, outgoings and profit.

Candidates will need to: Work with others to plan and carry out research to investigate how similar

companies have produced a solution (The company does not necessarily have to work with bands and artists or be a promotions company.)

Clearly record and display your findings Recommend a solution that will address the requirements of the task Produce a design brief, incorporating timescales, purpose and target

audience

18

ICT Task (cont.)Produce a solution, ensuring that the following are addressed:

it can be modified to be used in a variety of situations it has a friendly user interface it is suitable for the target audience it has been fully tested.

You will need to: Incorporate a range of: software features, macros,

modeling, and validation checks (used appropriately). Obtain user feedback Identify areas that require improvement, recommending

improvement, with justification Present information as an integrated document Evaluate your own and others’ work.

19

England: AS and A Levels - Changes to Assessments A broad range of question types to assess a wide range of skills Questions requiring extended answers give students the

opportunity to demonstrate the full breadth and depth of their understanding

Synoptic assessments that will test students’ understanding of the subject as a whole, including ability to: Develop a broader and deeper understanding of the

connections between the knowledge and understanding set out in the specifications as a whole

Draw together some of the key insights from their studies Make purposeful use of these in undertaking assessment tasks

Introduction of an extended project as a separate qualification for A level students to add to their study program.

20

7. Build and Develop Professional Capacity

Example: Teacher collaboration on assessment design and marking, Alberta, Canada

Test design: Identify student characteristics Assist in exam blueprint document Ensure curricular “fit” of the exam Pilot prototype multiple-choice and written response

forms Help develop writing assignments and their scoringItem building: Item development sessions held throughout the

province involve teachers in the creation of new multiple-choicce and written responses.

21

Teacher Collaboration – Alberta, Canada

Reviewing Tests: Each new examination form is reviewed by a

committee that includes classroom teachers. The committee examines both the written response

and multiple choice sections to ensure that the examination is fair, and demonstrates fidelity to the curriculum.

Confirming Standards: Before written responses are marked a committee of

teachers meets with examination branch staff to select student work for use in marker training (example papers, training papers and reliability review papers).

22

Teacher Collaboration – Alberta, Canada

The Marking Process: Each January and June approximately 1300

teachers meet to mark the written responses in a process with tight calibration to produce consistency in scoring.

Examination Advisory Committee: Once yearly representatives of various stakeholder

groups, including the Alberta Teachers’ Association meet with branch staff to review examination results from the previous school year, and to offer suggestions for the improvement of the examination program.

23

Scoring of Assessments - Ontario

Grade 9 Mathematics – raw score points = 46% multiple-choice, 54% open-response

Ontario Secondary School Literacy Test – raw score points = 48% multiple-choice, 52% open-response

Multiple choice items are machine scored Written open-ended responses systematically scored by qualified scorers

(mostly Ontario educators) Use of generic rubric to describe levels of performance, item-specific rubric

for each item, and anchors to illustrate descriptors for the score points to the rubric

Committee of educators define range of performance within each code of scoring rubric – ‘range finding’ development

Every scorer participates in extensive training System of checks and monitoring applied throughout process Quality assurance and statistical techniques to validate interrater reliability

and scoring validity Use of equating process to place student scores on common scale over two

adjacent years and to determine cut scores

24

Teaching-Learning Critical Pathway.Teaching-Learning Critical Pathway.

Gathering EvidenceGathering Evidence

Area of Greatest NeedArea of Greatest Need

Current PracticeCurrent Practice

Rubric, Data Wall, Culminating TaskRubric, Data Wall, Culminating Task

Professional Learning Community ActionProfessional Learning Community Action

Moderated MarkingModerated Marking

Share Findings, CommunicationShare Findings, Communication

Using Assessments to Inform Improvements:Example: Ontario

25

8. Are part of a clear and flexible learning trajectory across subjects and over education/career pathways

Example: Exam boards in England ensure consistency of standards within subjects while allowing flexibility of approach across subjects.

Five exam boards - governed by specifications, criteria, frameworks and oversight of Office of the Qualifications and Examinations (OFQUAL) Regulator

Exam boards: set out the course syllabus & specifications to meet

criteria set by the regulator oversee writing of exam questions and how exam papers

should be marked check coursework/controlled assessments provide training and quality control for examiners,

including use of ‘standardization meetings’ to compare marking across examiners

set grade boundaries to align marks and standards

26

National Qualifications Framework, England:Learner Pathways

27

9. Support data-informed decision-making for local and system improvement

Assessment for Learning:School and Classroom(Assessment Reform Group,1999)

Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)

Assessment for learning should be part of effective planning of teaching and learning.

Assessment of and for learning, plus relevant contextual, student and school data, is part of effective strategic and operational planning for state/province, district, school and classroom actions for improved teaching and learning.

Assessment for learning should focus on how students learn.

Assessment for learning focuses on how systems, schools and students improve their progress, achievements and outcomes.

28

Assessment for Learning:School and Classroom(Assessment Reform Group,1999)

Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)

Assessment for learning should be recognised as central to classroom practice.

Data-informed decision making, including assessment for learning, is recognised as central to educational practices at all levels of the system – province/state, district, school and classroom.

Assessment for learning should be regarded as a key professional skill for teachers.

Skill and capacity to use, understand and apply data to inform improved actions and outcomes should be regarded as a key professional skills for all educators, including teachers, principals, district staff and state/provincial officials.

Assessment for learning should be sensitive and constructive because any assessment has an emotional impact.

Careful consideration of which data to use, by whom and for what purpose is required to ensure the sensitive and constructive use of data, including assessment, to support improvement for all not to rank or judge unfairly.

29

Assessment for Learning:School and Classroom(Assessment Reform Group,1999)

Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)

Assessment for learning should take account of the importance of learner motivation.

The use of data, including assessment for learning, to generate motivation to improve should place an emphasis on developing respectful partnerships to engage educators at all levels of the system to work together.

Assessment for learning should promote commitment to learning goals and a shared understanding of the criteria by which they are assessed.

Shared goals for improvement, and indicators of success towards these goals, are developed and understood to generate a common commitment to improvement targets and learning outcomes.

Learners should receive constructive guidance about how to improve.

Building capacity for improvement involves providing feedback, strategies, resources and supports to enhance both professional learning and student learning.

30

Assessment for Learning:School and Classroom(Assessment Reform Group,1999)

Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)

Assessment for learning develops learners’ capacity for self-assessment so that they can become reflective and self-managing.

Data-informed approaches involve combining assessment of and for learning to balance external and internal accountability, while building the professional capacity for educators to implement self-evaluation, improvement planning and monitoring strategies.

Assessment for learning should recognise the full range of achievements of all learners.

Assessment for learning, and the use of related data, recognises the full range of achievements of all learners and schools, including not only achievement results but also progress over time and equity of outcomes for closing gaps in performance while also raising the bar overall.

31Students

School Boards/Authorities

Teachers, Principals & Administrators

FundingPolicy/ProgramsPublic Reporting

Ontario School Information System (OnSIS)

Staging Area

Board Validation & Sign-off by Director

MinistryValidation

Encryption&

Depersonalization

Elementary Secondary Data

Warehouse (ESDW)

Statistics Canada

Statistics Canada

EQAOEQAO

Other Sources

Other Sources

Data

Data

Data Data repository or data warehouse

Data repository or data warehouse

Student Management Systems (SMS)Student Management Systems (SMS)

Decision Support System

Decision Support System

EQAO

Stats Can

Other

Board Administrators

Pro

vin

ce-W

ide

In

form

atio

n

Data

Data

Data

MiSAGiARE

Info Info Info

Ministry of Education

32

10. Place quality of assessment over quantity

Examples: Australia – National standards (underway)

State Syllabi, State and Local Assessments(3 grades plus high school)

Canada – Provincial standards, syllabi & assessments(2 grades plus high school)

England – National standards & curriculum School-based assessments + national tasks (1 grade plus high school)Five secondary examination boards

Singapore – National standards and curriculumNational exams + school-based tasks(1 grade plus high school)

33

Common Practices Across Countries

Assessments are part of a tightly integrated system of standards, curriculum, instruction, assessment, and teacher development at the state or national (in small countries) level

Assessments include evidence of actual student performance on challenging tasks that evaluate 21st century standards of learning

Teachers are integrally involved in the development and scoring of assessments

Assessments are designed to continuously improve teaching and learning

Use of multiple-choice testing limited and combined with other forms of assessment

Emphasis on criterion-referenced and curriculum-embedded assessments

Not same approach for every student, every grade, every year Increasing use of teacher assessments in national reporting,

which external summative assessments in specific years/transition points

34

PrinciplesProposals Designed around a clear

vision of purpose and learning goals

Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies

Start with the end in mind – clarify and confirm vision for US high school graduates as engaged citizens, active contributors within the global economy and future leaders. Requires combination of functional skills, generic and specialist knowledge, performance competencies, and personal learning strategies

Integrate assessment design with common standards for consistency of curriculum expectations and clarity of teaching and learning standards. Involves criterion-referenced and curriculum-embedded assessment items. Attention to consistency within curricular areas. Vertical alignment over grade levels to support progression of learning.

35

PrinciplesProposals Encompass and combine

assessment of, for and as learning

Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning

Develop an assessment system focused on learning where summative assessments are one element of a wider repertoire of assessment information. Increase use and consistency of teachers’ assessments in reporting and monitoring student achievement and progress.

Use range of item-types and assessment approaches, including use of performance assessments, extended writing and project work. Use, but limit, time on standardized multiple-choice items and test taking.

36

PrinciplesProposals Embody high expectations to

engage, motivate, support and stretch all students to progress and achieve

Build and develop professional capacity

Range of assessments, including items and tasks that involve authentic experience, critical thinking and applied performance of students. Attention to student personal learning plans and progression targets. Growth and cohort analysis. Develop capacity for student self- and peer-assessment, including group work.

Educator capacity building is critical and requires time, resources (human, financial, material) and support. Teacher moderation is a powerful strategy to support teaching and learning throughout school year. Assessment items, constructs, interval data and other technical advice are required to be available and accessible to identify linkages between assessments, curriculum expectations, instructional strategies and interventions for specific students.

37

PrinciplesProposals Assess a range of content,

knowledge, skills and performance in authentic, applied and appropriate ways

Are part of a clear and flexible learning trajectory across subjects and over education/career pathways

Implement repertoire of formative and summative assessments with variety of approaches within and across grade levels to demonstrate knowledge, understanding and application.

Support clarity of learning pathways through high school and into college and career readiness, with accreditation options and credit accumulation requirements to indicate when a student is on or off track for success

38

PrinciplesProposals Support data-informed

decision-making for local and system improvement

Place quality of assessment over quantity

Implement data infrastructure (technology and systems) and culture (human capacity and data access) to enable collection, analysis and use of data at all levels of system

Shift emphasis from multiple-choice standardized tests as main approach for every student, every grade, every year. Combine external tests with coursework and other approaches to assess learning over time and in applied contexts. Develop consistency across assessment aligned with common standards, quality assurance mechanisms, and attention to building consistency in educator understanding and application of assessment strategies, including teacher moderation.