Benő Csapó http://www.staff.u-szeged.hu/~csapo/

Current Trends in Educational Assessment in Hungary:

Research and Development

Prague, 26 November 2014

UNIVERSITY OF SZEGED INSTITUTE OF EDUCATION

Outline

• Current trends in educational assessment

• Educational assessment in Hungary

• Developing an online diagnostic assessments system– a large-scale project at the University of Szeged

• Further plans to develop the educational assessment in Hungary

Function of assessment: providing feedback

“If you can not measure it, you can not improve it.” Kelvin

“Measure what is measurable, and make measurable what is not so.” Galileo Galilei

Current trends in educational assessment

What is new?

(1) New constructs

(2) New contexts

(3) New instruments

(1) Assessment of new constructs “21st Century Skills”

• Formative and diagnostic assessment• Embedded in learning and teaching• Based on a better understanding of

learning and development• Detailed, frequent• Provides immediate feedback• Helps to determine the directions of

intervention

(2) Assessments in new contextAssessment for learning

• Innovative item formats, improved validity

• Improved objectivity and reliability

• Improved precision

• Automated scoring

• Immediate feedback

(3) New InstrumentsTechnology-Based Assessment

TBA in PISA

• 2006, major domain: science– Technology-Based Assessment of Science– international option

• 2009, major domain: reading– Electronic Reading Assessment– international option

• 2012, major domain: mathematics– Computer-Based Problem Solving– innovative domain

• 2015, major domain: science– Computer-based assessment at all domain– Collaborative problem solving as innovative domain

USA: Assessment in the “Race to the Top”

• “Adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global economy”

• “Building data systems that measure student growth and success, and inform teachers and principals about how they can improve instruction”

Characteristics of the assessment projects in “Race to the Top”

• Two grant categories– Comprehensive Assessment Systems– High School Course Assessment

• Coalition of states• Two large coalitions received approximately

$330 million• Partnership for Assessment of Readiness for

College and Careers (PARCC) $170 million• SMARTER Balanced Assessment Consortium

(SBAC) $160 million

Educational assessment in Hungary

History and context: Two research and development communities

• Assessment centers at Budapest– National Educational Institute (-1990)

• Hungary participated in the first IEA project in the 1970s and 1980s

– After 1990: Several reorganizations– Today: Educational Authority

• International and national assessment programs

• Research and development centers at the University of Szeged – Test development projects from the 1970s– Several research groups established in the 1980s and 1990s – Today

• Research Group on the Development of Competencies (1996-)– Hungarian Educational Longitudinal Program

• Center for Research on Learning and Instruction– Developing Diagnostic Assessments

– Training of educational assessment experts (1991-)

Educational Assessments in Hungary

PIRLS TIMSS PISA

Diagnostic Assessments

Assessment of Basic Competencies

Grade

Matura(Abitur)

Diagnostic Assessment System

Characteristics of the Assessment of Basic Competences

• Reading and mathematics are the assessment domains

• Every student is measured at grade 6, 8 and 10

• The data are longitudinally connected

• Sophisticated value-added models are used

• School-level results are published (‘soft accountability’)

Developing Diagnostic Assessments

A project implemented by the Center for Research on Learning and

Instruction University of Szeged

Two phases: 2009-2011, 2012-2014

edia.hu

http://edia.hu

English

http://edia.hu/?q=en

Publications / Books

Demo test

Online Diagnostic Assessment System

• A complex system for supporting learning and teaching• Based on the analysis of learning and developmental

processes, and the outcomes of learningSee -> framework development

• An online accessible assessment platform See -> eDia

• An item bank containing thousands of items in innovative formatsSee -> demo test for item formats

• Teacher trainingpreparing the teachers for the utilization of the system

• Developing intervention materials (e.g. computer games) for enhancing students’ skills if needed

Devising assessment frameworks for reading, mathematics and science for

the first six grades

How an assessment framework looks like?

PISA TIMSS Diagnostic

- application - content - content (literacy) - reasoning - reasoning

- application - application

A three-dimensional model for framework development

External, social expectations

GENERAL ABILITIES

Disciplinary content

Internal, psychological

LITERACY

CONTENT KNOWLEDGE – EXPERTISEPISA, three main literacy domains

Curriculum-based assessment, 1st and 2nd IEA Science and Mathematics Study

There may be unexploited

potentials here

Devising assessment frameworks:working methods and products

• Broad national and international cooperation

• Results published in 2x3 volumes (in English and in Hungarian)

• Opening theoretical chapters (international co-authors)

• Detailed standards by Hungarian expert groups

ReadingEditors

Benő Csapó and Valéria Csépe

           

     

Pyschology Literacy Discipline

Leo Blomert Wolfgang Schnotz Krisztián Józsa

Valéria Csépe Edit Katalin Molnár János Steklács

MathematicsEditors

Benő Csapó and Mária B. Szendrei

           

     

Reasoning Literacy Discipline

Terezinha Nunes Lieven Verschaffel Mária B. Szendrei

Benő Csapó Csaba Csíkos Júlia Szendrei

ScienceEditors

Benő Csapó and Gábor Szabó

           

     

Reasoning Literacy Discipline

Philip Adey Erzsébet Korom Gábor Szabó

Benő Csapó Mária B. Németh Erzsébet Korom

Result of the Framework

Development

Three volumes, published both in Hungarian and in English

May be downloaded in .pdf

The eDia platform

• The system runs at the servers of the University of Szeged

• Item builder module• Item bank unit• Test delivery module• Data bank• Data analysis module• Feedback module

The item development process

• Training the item writers (350+)– assessment theory and item writing

– content of the assessment (frameworks)

– ICT and the eDia platform

• Item writing

• Reviewing by the group leader

• Supervising by the „dimension chief”

• Piloting

• Determining the item parameters

Innovative item formats

• Text and voice for the grades 1-3• Stimulus

– text, voice– picture, animation– video– simulation

• Response capture– everything what is possible on paper, keyboard,

mouse– manipulating and moving objects on the screen, drag

and drop– touchscreen solutions

Characteristics of the Online Diagnostic Assessment System

• Serving ca. 600 000 students (grades 1-6)• Communicating with ca. 20 000 teachers• Making the service available in ca. 3 000

schools• Managing ca. 10 000 test items• Maintaining and analyzing students’ data• Providing both students and teachers with

sophisticated feedback in user friendly format

Assessments of the major domains

• Reading, mathematics and science are the major domains

• Each major domain is measured in 3 dimensions in different scales

• Large item banks are used to measure the major domains

Assessment of minor domains

• There are 16 minor domains in the system– school readiness– writing skills, musical abilities, English as a second

language– health literacy, financial literacy, visual skills, civic

competencies– combinatorial reasoning, inductive reasoning– problem solving, learning to learn, ICT literacy,

creativity– social skills, motivation

• Test batteries are prepared for the minor domains

Plans for the next phase: 2015-

• Online developmental exercises for helping slow students to catch up

• Gamification• Visualization• Integration with the regular instructional

processes• Extending to the kindergarten population

The significance of early education(The Heckman curve)

The economists’ view

The developmental psychologists’ view:

Solid foundation for later learning

Basic skillsInterestAttitudes

Motivation

Can online assessments be used in kindergarten?

YES!

Yes!

Further plans to develop educational assessment in

Hungary

Digitizing the Assessments of Basic Competencies

• Present problems– only reading and mathematics is assessed – expensive– slow feedback– difficult to interpret

• Plans for TBA– adding science– developing item banks– immediate feedback– visualizing the data to help the interpretation

Technology-based reform of the ‘matura’

• Present problems– too many elective subjects– low objectivity– no comparable scales– etc.

• Plans for the reform– technology based assessments only– 5 compulsory subjects: Hungarian language

and literature, history, mathematics, science, English (no electives)

Conclusions

• Technology-based assessments work well in kindergarten and primary schools

• Technology can be extended to other domains and age groups, the precision, objectivity and reliability can be improved

• Further work is needed to integrate the assessments into the regular classroom activities and educational processes

www.staff.u-szeged.hu/~csapo

Thanks for your attention!