Measuring of student subject competencies by SAM: regional experience

Elena Kardanova

National Research University Higher School of Economics

Outline of presentation› SAM based model for assessment of student subject

competencies

› Regional diagnostic study: sampling and procedures of test administration

› SAM regional norms and presentation of results

› Interpretation and uses of SAM information: primary analysis of factors that influence educational results in primary school

› SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school

› SAM experience in other countries

SAM based model for assessment of student subject competencies

Approaches to results interpretation

Norm-Referenced

The result of individual student is interpreted depending on the achievement of the whole population

Each student gets test score

Norms are set

Approaches to results interpretation

Criterion-referenced

The gradual option of achievement scale is developed. It’s based on students integrated test scores and benchmarks that divide all participants of testing into groups that relevant to different proficiency levels

Each participant is assigned a proficiensy level

Students estimation

› Rasch model is used as a test model

› Test scores are reported on a 1000-point scale with a mean at about 500 and standard deviation of 50

› Test scores of all participants are on the same metric scale regardless of the time of test administration and specific set of test items completed

Mathematical competence scale

Proficiency level 3

We expect student A to successfully complete at least 50% of level 3

items

Student A

Proficiency level 2

Proficiency level 1

Below level 1

We expect student B to successfully complete at least 50% of level 2

items

Student B

We expect student C to successfully complete at least 50% of level 1

items

Student C

We expect student D to be unable to

successfully complete even 50% of level 1

items

Student D

500

570

430

Items of

the 3rd

level

Items of

the 1st

level

Items of

the 2nd

level

Estimation of examinees

Interpretation of benchmarks

Benchmarks:

570 (border btw. 2 and 3 proficiency

levels)

500 (border btw. 1 and 2 proficiency

levels)

430 (border btw. 0 and 1 proficiency

levels)

Regional diagnostic study: sampling and procedures of

test administration

Velikiy Novgorod and its area

Regional diagnostic studyMay 2012 Sample size: 4406 students of 4-th grade (the region’s whole population of fourth grade students )

No selection at the school or classroom level

Description of research sampling

47% boys,

53% girls

72% urban,

28% rural

Number of students 4406

Number of schools 189

Number of classes 297

Number of settlements 134

Regional diagnostic study: procedure

› Paper&pencil form

› Administration to the whole class by the teacher

› Two 45-minute testing sessions with a 15-minute break

› Whole region in 1 week

SAM regional norms and presentation of results

SAM results

Integrated test score( relation of result to the metric scale)

Proficiency level (relation of result to grade scale)

3D profile (relation btw. results of 3 subtests)

Normative-referenced interpretation: Statistical norms (Mathematics)

Average group norms Mean Standard deviation

517 34

Socio-cultural norms Mean

561

Average individual norms Mean Standard deviation 522 49

Percentile individual norms 10th percentile 90th percentile 459 581

Math profile for a sample of students

Distribution of test participants on proficiency levels (Mathematics)

Distribution of students of different schools of the region at proficiency

levels (mathematics)

› Schools put in order by increasing of the mean test score

› For every school the nean test score is indicated in brackets.

Distribution of students of different classes within the same school by achievement levels (mathematics)

Interpretation and uses of SAM information: primary analysis

of factors that influence educational results in

primary school

Key questions

1. What is the efficiency of different educational programs?

2. How different factors influence on students learning?

3. What characteristics of learning environment influence on educational quality in primary school?

4. How teachers and school work can be improved?

Educational environment and its characteristics that can be

examined

Level Responsibl

e entity

Domains

Regional /

Federal

Federal or local Government

Regional educational policy, Federal educational standard, unified exams, curriculum

School School principal

School policy, type of school, curriculum, condition of building and classes, sports sections, school activities, etc. Recruiting of teachers and administrative personnel

ClassTeacher Quality of teaching, methods of teaching,

pedagogical approaches. Quality of students feedback. Educational tasks and goals

Outside of

school,

family

Parents and student

Out-of-school activities, additional education, social-economic status, parents education, books, computer, Internet access, personal motivation

Information sources for analysis of factors

Contest questionnaries (for teachers, for administrative personnel)

› Set of contest characteristics can vary depending on regional research tasks

› Focus at characteristics of school environment that can be corrected to improve the quality of education

Technical informationThe detailed information about school learning and teaching features can be collected purposefully (e.g. educational programs).

Relation btw. SAM math results and type of educational institution

› There are 15% of student study in gymnasium.

› Differences btw. schools are statistically significant: gymnasiums get better tests results.

› Number of children at 2nd level is the same. Difference is btw. children at 1st and 3rd levels.

Relation btw SAM math results and type of settlement

Percentage of children at 3rd level is higher in the city and decreases in towns and villages. Its vice versa for percentage of children at 1st level – it’s higher in villages.

Relation btw SAM math results and type of settlement

Most of students are on the 2nd proficincy level In Velikiy Novgorod the results are slightly better – bigger percentage of children is at 2nd and 3rd levels.

Relation btw SAM math results and size of the class

› We can single out 2 types of classes – big and small

› Small classes are those that have less than 11 students, big classes have 11 and more students (maximum number of students in one class is 33)

› All together we analyzed 76 small and 152 big classes

Relation btw SAM math results and size of the class

Results of children in small and big classes are not statistically different.

SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school

Pedagogical approaches› Currently it is widely assumed that teachers’ beliefs

about the nature of teaching and learning include both “direct transmission beliefs about learning and instruction” or, so called, “traditional beliefs” and “constructivist beliefs about learning and instruction” (OECD, 2009).

› 2 educational approaches: traditional and constructivist

– The traditional approach implies that teacher communicates

knowledge in a clear and structured way, explains correct solutions, gives learners clear and resolvable problems and ensures peace and concentration in the classroom

– The constructivist approach implies that students are active participants in acquisition of knowledge, students’ own inquiry is stressed developing problem solutions

Teachers survey

› Special teachers questionnaire›228 teachers in total›56 teachers work in Velikiy Novgorod, and 172 in Novgorod region›17 teachers work in gymnasium and 211 in comprehensive school›186 teachers graduated from university and 42 graduated from college›Work experience varies from 2 to 48 years. Average is 25 years

Correlation -,204** is significant at the 0.05 (2-tailed)

Traditional

Constructivist

Teachers’ general pedagogical approach

Clusterization of classes

Pedagogical approaches

› Constructive approach positively relates to results of learning in math and Russian language: the higher level of constructivism of a teacher the higher test scores students have.

› Constructive approach positively relates to the number of students in class at 3rd level and negatively with number of students at 1st level and below 1st level.

› Traditional approach doesn’t have significant relation with learning results – neither with test scores or distribution of children at levels.

SAM experience in other countries

› Kazakhstan

› Kyrgyzstan

› Tajikistan

Thank you for your attention

Elena Kardanova

ekardanova@hse.ru

Center for monitoring and quality of education

Institute of education

Higher School of Economics

http://ioe.hse.ru/monitoring/