Reporting International Test DataCh. 3 Measuring Civic Knowledge

and Understanding

ICCS NRC meeting

Madrid, Feb 2010

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge

• Instrument

• Described scale

• Results

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge - Instrument

• 80 item (reduced to 79 - CI2HRM2 removed)• 73 MCQ, 6 constructed response• 62 new, 17 CIVED trend• Items showing extreme item by country

interaction (>1.5 logits) removed from scaling for estimates of test scores in individual countries

• Report to include example release items illustrating range of item types and content (including % correct by country and scale location)

NRC MeetingMadrid

February, 2010

X 2 X

XX XX XX XX

XXX XXXX

XXXXX 1 XXXXX

XXXXXXX XXXXX

XXXXXX XXXXXXX

XXXXXXXX XXXXXXXXX

XXXXXXXXXX 0 XXXXXXXX

XXXXXXXXX XXXXXXXX

XXXXXXXXX XXXXXXXXX XXXXXXXXX

XXXXXXXX XXXXXXXX

XXXXX -1 XXXXX

XXXXX XXXX

XXX XXX

XX X X

-2 X X

14 25 5 37 46 75 2 4 49 56 74 51 71 6 9 28 55 59 77 10 32 34 68 19 27 44 20 30 33 36 65 72 26 40 41 50 53 57 58 61 70 11 16 18 21 23 31 64 66 69 12 17 35 76 78 29 3 38 43 47 63 7 13 15 22 48 62 67 79 1 52 54 42 60 73 24 39 45 8

• rp = 0.62 (relative shift item difficulties = ln(0.62/0.36) ≈0.49 logits

Measuring Civic and Citizenship Knowledge – Described Scale

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Described Scale

• Item map (ordered item descriptors)

0.70 Justifies the use of the separation of powers between the judiciary and the parliament.

0.70 Evaluates a risk of violent protest action in terms of possible outcomes.

0.67 Integrates government accountability through FOI legislation with the capacity of citizens to keep themselves informed.

0.56 Describes national elections as an element of "democracy".

0.53 Justifies the opposition to movie censorship as a possible restriction of freedom of expression.

0.5 Integrates the responsibility of citizens to obey the law with the right of citizens to vote in elections.

… …

-0.19 Illustrates with an example the responsibility of voters in democracy.

-0.19 Interprets that a statement expressing a position about "flags" and "anthems" is an opinion rather than a fact.

-0.2 Relates equity and social justice to the provision of services by governments without charge.

-0.37 Generalizes the impact of environmental degradation across to the whole world.

-0.4 Generalizes learning the main language in a country to potential for community participation.

-0.4 Describes the applicability of the universal declaration of human rights in terms to all people.

… …

-0.86 Relates the responsibility of citizens to obey the law with the right of citizens to vote in elections.

-0.96 Relates equality and inclusiveness to equity as necessary conditions for accepting people who have served punishment for crimes in the community.

-0.97 Relates an example from a community setting to the concept of equity.

-1.19 Hypothesizes about the relationship between the secret ballot and freedom of voter choice.

-1.77 Identify informing consumers as a key aspect of promoting ethical consumerism

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Described Scale

• Item map (ordered item descriptors)

3 0.70 Justifies the use of the separation of powers between the judiciary and the parliament.

3 0.70 Evaluates a risk of violent protest action in terms of possible outcomes.

3 0.67

Integrates government accountability through FOI legislation with the capacity of citizens to keep themselves informed.

2 0.56 Describes national elections as an element of "democracy".

2 0.53 Justifies the opposition to movie censorship as a possible restriction of freedom of expression.

2 0.5 Integrates the responsibility of citizens to obey the law with the right of citizens to vote in elections.

… … …

2 -0.19 Illustrates with an example the responsibility of voters in democracy.

2 -0.19 Interprets that a statement expressing a position about "flags" and "anthems" is an opinion rather than a fact.

2 -0.2 Relates equity and social justice to the provision of services by governments without charge.

1 -0.37 Generalizes the impact of environmental degradation across to the whole world.

1 -0.4 Generalizes learning the main language in a country to potential for community participation.

1 -0.4 Describes the applicability of the universal declaration of human rights in terms to all people.

… … …

1 -0.86 Relates the responsibility of citizens to obey the law with the right of citizens to vote in elections.

1 -0.96

Relates equality and inclusiveness to equity as necessary conditions for accepting people who have served punishment for crimes in the community.

1 -0.97 Relates an example from a community setting to the concept of equity.

B1 -1.19 Hypothesizes about the relationship between the secret ballot and freedom of voter choice.

B1 -1.77 Identify informing consumers as a key aspect of promoting ethical consumerism

395

479

563

L1

L2

L3

B1

0.8 logits

NRC MeetingMadrid

February, 2010

300

350

400

450

500

550

600

650

700

750

Measuring Civic and Citizenship Knowledge – Described Scale

2 Items

22 Items

16 Items

39 Items

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Described Scale

Level 1

Students working at Level 1 demonstrate familiarity with equality, social cohesion and freedom as principles of democracy. They relate these broad principles to everyday examples of situations in which protection of or challenge to the principles are demonstrated. Students also demonstrate familiarity with fundamental concepts of the individual as an active citizen: they recognise the necessity for individuals to obey the law; they relate individual courses of action to likely outcomes; and they relate personal characteristics to the capacity of an individual to effect civic change.

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Described Scale

Level 2

Students working at Level 2 demonstrate familiarity with the broad concept of representative democracy as a political system. They recognise ways in which institutions and laws can be used to protect and promote a society's values and principles. They recognise the potential role of citizens as voters in a representative democracy, and generalise principles and values from specific examples of policies and laws (including human rights). Students demonstrate understanding of the influence that active citizenship can have beyond the local community. They generalise the role of the individual active citizen to broader civic societies and the world.

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Described Scale

Level 3

Students working at Level 3 make connections between the processes of social and political organisation and influence, and the legal and institutional mechanisms used to control them. They generate accurate, hypotheses on the benefits, motivations and likely outcomes of institutional policies and citizens' actions. They integrate, justify and evaluate given positions, policies or laws based on the principles that underpin them. Students demonstrate familiarity with broad international economic forces and the strategic nature of active participation.

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Described Scale

Level 1: broad concepts; “big ideas”; mechanistic

Specificity

Level 2: interconnectedness of systems; wider sphere of citizen influence

Evaluation

Level 3: holistic; strategic

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Results

Comparison of means

• Means range from 380 to 576– Levels 1 to 3

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Results

Comparison of means

Multiple comparison

Country

Finland ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Denmark ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Korea, Republic of ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Chinese Taipei ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Sweden ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

P oland ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Ireland ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Switzerland † ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Liechtenstein ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Italy ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Slovak Republic² ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Estonia ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

England ‡ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

New Zealand ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Slovenia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Norway † ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Belgium (Flemish) ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Czech Republic † ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Russian Federation ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Lithuania ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Spain ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Austria ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Malta ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Chile ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Latvia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Greece ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Luxembourg¹ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Bulgaria ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲

Colombia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲

Cyprus ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲

Mexico ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲

Thailand ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲

Guatemala ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲

Indonesia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲

P araguay ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲

Dominican Republic ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼

▲ Average achievement signif icantly higher than in comparison country

▼ Average achievement signif icantly low er than in comparison country

† Met guidelines for sampling paticipation rates only after replacement schools were included.

‡ Nearly satisfied guidelines for sample participation only after replacement schools were included.

¹ Survey administration did not follow international guidelines.

² National Desired Population does not cover all of International Desired Population

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Results

LevelLevel boundary

Items (N)

Items (%)

Students All (%)

Students Max (%)

Students Min (%)

3 16 20 28 58 1

>563

2 39 49 31 39 7

>479

1 22 28 26 44 10

>395

B1 2 3 16 61 2

Comparison of levels

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Results

Level No. countries with max % students in level

3 14

2 14

1 8

B1 2

Comparison of levels

NRC MeetingMadrid

February, 2010

Measuring Civic and Citizenship Knowledge – Results

Level No. countries > 60% students in levels

2- 3 23

B1 - 1 7

Comparison of levels

NRC MeetingMadrid

February, 2010

Gender differences in achievement• International means

– Girls 511– Boys 489 – Difference is statistically significant

• Mean for girls higher in all countries– Statistically significantly higher in 32 countries

• Range of mean differences 2 to 48 (median =22)

Measuring Civic and Citizenship Knowledge – Results

NRC MeetingMadrid

February, 2010

Trend Comparisons

• Calculated only on the basis of 17 trend items (different metric to ICCS)

• Dimensionality conflating results for complete scaling from both ends– CIVED through representativeness of

trend items– ICCS through broadening of framework

and constructs

Measuring Civic and Citizenship Knowledge – Results

NRC MeetingMadrid

February, 2010

Comparisons over time

• 17 countries used same translations of CIVED items for comparison

• CIVED population mean = 500

• ICCS population mean = 484– Difference is statistically significant

5 countries (stat sig one country)

11 countries (stat sig six countries)

Measuring Civic and Citizenship Knowledge – Results