THE WORLD BANK Development, Purposes, Uses of EGRA Session 1.1 1

THE WORLD BANK

Development, Purposes, Uses of EGRA

Session 1.1

1

Outline

1. Purpose: why Early Grade Reading (Assessment)?

• More focus on quality

• Using Early Grade Reading as a “marker” of quality

2. Development

3. Uses

2

Outline




2. Development

3. Uses

3

Purpose: Why? Quality issues

■ What are the big international goals?

■ How do low income countries compare to high income countries? LI to HI ratio

− Gross primary enrollment: 95%− Net primary enrollment: 80%− Gender parity NER: 94%− Completion: 58%− Learning achievement: Approx 30%?− Learning achievement: Median LI = 3rd percentile

of HI or lower

LI= Low income, HI = High income4

Just as interesting is the internal distribution within the poor countries, and the comparison with the rich countries ...

5

PIRLS 2006 Results

0

100

200

300

400

500

600

700

0 20 40 60 80 100

Sc

ore

Percentiles

OECD curve extrapolated to show the "0th" percentile

South African medianis below the "0th" percentile in OECD

SA, Morocco, Indonesia

SA

6

PIRLS 2006 Results

0

100

200

300

400

500

600

700

0 20 40 60 80 100

Sc

ore

Percentiles

OECD curve extrapolated to show the "0th" percentile

Also notice the much greater inequalityin poorer countries (recall that it is not just the gap that matters, but therelative gap!!

2.4 to 1 ratio

1.2 to 1 ratio

S. Africa, Morocco, Indonesia

7

0

100

200

300

400

500

600

700

800

0 20 40 60 80 100

Percentiles

Sco

re

SD at the "worst" end: 88, Coeff of Var = 0.23

SD 44, Coeff of Var = 0.07

Countries

PIRLS 2006 Results

8

9

In path of development of good performance, the biggest and fastest drop is in the number of worst performers…

In the “standard” or better path, an educational middle class is developed by helping those with worst performance…

And the drop in those with very poor performance is much bigger than the increase in those with good performance

But this does not happen by accident…

South AfricaMoroccoKuwaitQatar

Indonesia

Iran

SloveniaSlovak Rep.FranceDenmarkItaly

PIRLS 2006 ResultsP

erce

nt o

f le

arne

rs

0

10

20

30

40

50

60

70

80

90

Lowest Medium Highest

Reading competency levels10

11

Need to work at the worst end of the distribution, and with the countries that are really lagging…

Conclusion on quality issues

■ So, quality, and above all equality of quality (more equality in learning outcomes) seems to be lagging…

■ EFA goals do refer to quality

■ But most of the practical emphasis is on access, most of the judgment about “are we on track” is around access

■ Some moves afoot to track learning outcomes in a consistent way

■ And countries do participate in international and regional assessments, so we are not completely in the dark…

So, why a focus on Early Grade Reading?

12

If quality matters, why focus on Early Grade Reading?

Early Grade +Reading =

■ intervene early,

■ intervene on reading,

■ have some way to assess orally

Let’s see if we can motivate those conclusions

13

Why early?

■ Refer to the large and reliable literature on benefits of early (even ECD) intervention

■ But let’s focus on a few simple things

14

“For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath.”

Matthew Effect?

Why early?

15

Grade in years and months (thus 1. is 6 months into Grade 1)

6

Why early? Matthew Effect in reading

Data from the US

Children below a certain level by the end of Grade 1, stay behind forever, and the gap widens

And, if they cannot read, they fall behind in everything else

Initial SES gets amplified

Wo

rds

per

min

ute

16 Good, Simmons, Smith (1998)

Why early?

Reading Trajectories of Low and Middle Readers

Wo

rds

per

min

ute

Good, Simmons, Smith (1998)17

And continues to university & graduate school …

Kerchoff and Glennie (1999)

This controls for SES and prior academic achievement, so it is the NET effect of a demanding curriculum.

12th Grade Schooling

12th Grade Achievement

Attainment Four Years Later

Attainment 10 Years Later

LoSchool

LoCurriculum -4.883 .559 .470

HiCurriculum 7.047 -1.778 4.395***

MidSchool

LoCurriculum -6.108*** 2.054* .266

HiCurriculum 7.587*** .910 4.111***

HiSchool

LoCurriculum -5.138*** -.169 .453

HiCurriculum 7.393*** .451 5.481***

18

Why early?

■ Not only does the problem, if unresolved, amplify over time, but…

■ According to various experts, fixing a problem: in grade 1 takes 30 minutes in grade 5 takes 2 hours

■ Teaching good habits early is key, and it becomes very difficult to remediate later

19

Why reading?

■ No, it is not “the only thing that matters”

■ But it is a good one to start with

It is a (the?) foundational skill - Hard to imagine anything else going well if children

can’t read well and soon

It can be used as a marker - Hard to imagine a good school that can’t teach

children to read, if children are not reading, the school (district, country) needs serious help

20

Why oral reading?

Reason 1 Oral reading seems to be good

predictor

At least in the US (other industrial?) perhaps the best predictor, in early grades, of concurrent and even later success in a broad range of areas

21

Oral reading predictive powerExamples:■ Wilson (2005): 0.74 correlation between oral reading measures and broader cognitive

achievement in Arizona■ Fuchs, Fuchs, Hosp, and Jenkins (2001) survey and explain the rather high correlations between

oral reading fluency and a large variety of other tests.■ Vander Meer, Lentz, and Stevens (2005): 96% of children judged to be at risk using oral reading

turned out to be “non proficient” in the Ohio’s more comprehensive test, while of those classified as “low risk” using oral reading fluency, 72% were classified as proficient using a more comprehensive test.

■ Shaw and Shaw (2002): find similar results for the relationship between simple measures of oral fluency and deeper state-wide measures of reading in Colorado

■ Fuchs, Fuchs, and Maxwell (1988): correlation of 0.91 between oral reading fluency and other comprehensive tests

■ Juel (1988) : “The probability of remaining a poor reader at the end of fourth grade, given a child was a poor reader at the end of first grade, was .88 .... the probability of remaining an average reader in fourth grade, given an average reading ability in first grade, was .87.”

■ Schilling, Carlisle, Scott, and Feng (2007): 80% of at-risk with ORF turned out to be in bottom quartile with Michigan’s own reading test at EOG

■ Wood (2006) finds significant correlations between oral reading fluency and later curriculum-based tests, and finds that oral reading fluency adds explanatory value even when other factors are considered.

■ Some of these recommend adding comprehension and vocabulary (EGRA does comprehension , not vocabulary), but ORF by itself does a pretty good job

22

Why oral reading?

Reason 2:If use pencil and paper test too early: children bottom out, and test provides too little information on why children can’t read

23

Why oral reading? Reason 2

Case of country X■ Pencil and paper test■ Mostly attempt to directly measure comprehension

Sufficient: 15% Basic: 24% Below grade: 15% Less than below grade: 46%

■ We know they do not comprehend, but there seems to be little sense of why: can they dominate the basics?

■ Authorities admit that children bottom out if only try to measure comprehension, so try to measure simpler things:

24

About the easiest level:

25

■ If can’t do it, why?

■ Can’t decode?

■ Don’t have the vocabulary?

■ Some other problem?

■ Hard to say, it seems…

26


27


Elements of oral reading are in accord with curricular framework (at least for some countries) but frequently there are no specific (teacher-level) guidelines on how to assess…

28

29

Mother Tongue, Grade 1

But this almost never happens

30

31

Mother Tongue, Grade 2


■ Not in all (don’t know really how many – has anyone studied this?)

■ Even then maybe not specific enough (compare to any US state)? (How specific is optimal? How specific is “good enough”?)

■ In no country I’ve studied is it operationalized in terms of specific assessment standards or procedures that teachers and officials can actually use

So, at least in some countries the curriculum does get down to specifics, but…

32

Outline




2. Development

3. Uses

33

Development thus far

■ “Organic” process: meets “market test” at each step

■ First: informal, small samples, see if it was useful at generating awareness, very little funding

■ Attention attracted

■ Some funding to try it a bit more formally USAID funding: validate efforts thus far with expert opinion,

try some more applications

Experts validate, suggest increased formality, seriousness of trials

World Bank adds some funding, try it in two more international languages, local languages

34

Development thus far

■ Tried it in some more countries

■ Countries found it useful: “market test” met

■ Assess statistical properties, test further country reaction, do a few more countries, “market test” further met

■ And here we are: further evaluate results, discuss, chart further direction and uses

35

Outline




2. Development

3. Uses

36

Possible uses

■ Range of uses of the approach not predetermined

■ Seek guidance from all of you

■ And, different uses would require very different versions of the idea…

■ That being said, here are some of the uses that are possible…

37

Possible uses■ But… these uses have been tried or at least discussed

(and there is some sequentiality here):

Policy awareness and motivation Macro Community-based

Impact tracking and evaluation Project monitoring Project impact and evaluation System monitoring over time

Teacher-based assessment (Could link to community-based awareness, accountability?)

38

Possible uses

Country comparisons Within language groups? Not even?

Not a good use, according to many experts’ recommendations:

High-stakes bureaucratic accountability E.g., bureaucratic reporting for some high-stakes purpose

39

Documents

THE WORLD BANK Development, Purposes, Uses of EGRA Session 1.1 1