Approaches to evidence based policy making in education

Professor Geeta Kingdon

Congratulations CBSE

CAER commitment to evidence based decision-making

Centre will help to improve learning by promoting reliable and valid assessments

also help to create capacity do robust evaluations

Thus bring assessment-led reform in Indian education

Splendid example of a PPP

Talk is on evidence based policy, focus on macro

Current landscape

Pitiably poor learning levels (PISA, EI, NAS, ASER, NFHS, SchoolTELLS; NAS showed that <50% of class V kids could answer “how much greater is 555 than 198”? )

Measuring learning can be disheartening, when much effort & large resources expended (SSA reforms, RTE, NCF, 6 Pay Comm, TET, NCFTE, NCSL) & still outcomes are poor; denial

Encouraging that Planning Comm’s 12th FYP (GOI, 2013, p49) “will place the greatest emphasis on improving learning outcomes at all levels”

Steps for evidence based decision-making

Obtain the evidence (e.g. measure learning levels)

Evidence has to be credible (e.g. measure learning with validity and reliability, with proper sampling)

Embrace & analyse results to reach diagnoses

Formulating policies to take remedial action

Pilot test new policies, to see their impacts on learning

Scale up policies that have strong impact at lowest cost

Evidence from learning assessments – can empower parents

An imp potential use of evidence from A – is parental info

Has potential to increase school accountability

World university rankings, but not school rankings

Fear they may reflect socio-econ background but there are ways of reducing that

In India a debate needed

Evidence from assessment of teachers

One potential reason for poor learning - teachers themselves lack competence

Teachers rarely tested in large scale way

Evidence from T tests in India showed poor competence

Led to the decision to bring in the TET

SchoolTELLS survey

The teacher tests were graded by SCERT Bihar staff

Assessment tasks for teachers aligned with standard teaching tasks that teachers in primary school would be required to do in the classroom routinely.

Language tool helped to understand teacher’s ability in the following:

Do you know: e.g. meaning of difficult words in a grade 4 level text Can you explain : e.g. explain difficult words in simple language or

summarize a Std 4 story text effectively Can you spot common mistakes: e.g spelling and grammar mistakes

Maths tool also helped to understand the teacher’s ability in the following :

Do you know: e.g. solve problems Std 4 or 5 level Can you explain: e.g. explain problem solving in simple steps Can you spot and analyze common mistakes : e.g in arithmetic

operations

TASK 1: VOCABULARY RELATED TASKS

Grading done on 3 criteria:

Was the word meaning “meaningful” ? Was language used “easy to

understand” ? Were there any spelling mistakes ?

From Std 3 onwards, the vocabulary in language textbooks becomes difficult. So, teachers need to be able to explain difficult words in simple language.

TASK 2 : SUMMARIZING TEXTS

TASK 3: SPOTTING SPELLING & GRAMMATICAL MISTAKES

Deficits in teachers literacy skills

43% of word meanings correct; 57% wrong

45% of summaries were meaningful; 55% wrong

40% teachers did not have spelling mistakes (in a 2-sentence write up)

35% had 1-2 mistakes; 25% had >=3 mistakes

Only 50% of teachers could spot >3 mistakes in a write up in which we had deliberately introduced 6 mistakes.

TASK 6 : SPOTTING & UNDERSTANDING COMMON MISTAKES IN ARITHMETIC

Samples of children’s work shown. Teachers asked to choose…

Deficits in teachers’ numeracy skills

78% could spot correct one when presented a sample of three simple division sums

24.5% could do a percentage sum; 75% not

27.9% could do an area sum; 72% not

About 20% said they never had problems in addressing the maths queries of their pupils

How this evidence and TET evidence helps

School-TELLS (2008) & ‘Inside Classrooms’ study (2011) highlighted deficits in T knowledge & ability to teach

This evidence contributed to decision to introduce the TET

TET objective to vet applicants, ensure competence Abysmal pass rate - 0.4 to 3.7% pass rate

Evidence in this T assessment is extremely valuable

The evidence in this T assessment is extremely valuable It helps to identify the training needs of teachers.

It can inform policy makers who decide training curricula

But have states used this evidence in this way? States need the desire to analyse; make use of this evidence; a transparent approach

Congratulate the CAER for analysing C-TET – this is evidence based policy making (the policy maker CBSE sought evidence – gave data)

Is this evidence relevant only for govt and rural private schools?

Is this problem of low cognitive skills of Ts confined to government primary teachers ?

Clearly more generic problem – Even highly paid govt T have major deficits in skills TET evidence

Private schools cannot be complacent on this

Testing T can help to assess the training needs of each T

The importance of evidence

There are many initiatives to improve education

NFE (1982) OBB (1986) TLC (1988) MDM (1982) SK (1987) LJ (1988) DPEP (1993) SSA (2003)

Aadhar, ABL, MLE, Nallikali, Nai Disha, Read India, RTE

Have these programs had impact? – little evidence to judge

Efforts to improve more successful, when based on evidence

Culture of seeking evidence Why base decisions on evidence?

While poor quality schooling does not threaten lives, it seriously affects people’s quality of life, and even longevity.

In medicine, its unthinkable without thorough testing by experts, & the use of most robust, expensive randomised control trials; But in Education, Ministers freely make policies without consulting evidence / experts

In good educ systems, and ideally, policy will be not made on supposition, ideology or political expediency. Govt seeks evidence

UK policy makers’ use of evidence fromimpact evaluations

Early intervention - quality & effectiveness of pre-school experience in securing better long term outcomes was used to justify more investment (Sammons et al, 2006)

Class size - evidence of no stat difference was used to justify not extending the policy into later years (Blatchford et al, 2002)

Formative assessment - effect sizes in attainment (alongside pupil & teacher perspectives) led to inclusion in national policy (Wiliam et al, 2004)

Education Maintenance Allowance - evidence from pilot study of post 16 retention led to national roll out, though long-term sustainability was worse in pilot areas (Middleton et al, 2005)

Policies could be better, if evidence based

The narrative in the 6th Pay Comm for across-the-board doubling teacher salaries (without increased accountability) was : it will motivate teachers. Was this effective? Did it raise teacher effort? No one checked. [next slide]

A state govt recently announced it would regularise 176,000 para teachers, in the name of quality of education; it did not look at the relative effectiveness of regular and para teachers (3 papers)

Under RTE it has been made mandatory for teachers to have B.Ed. Certification, reduced PTR to 30, and many inputs mandatory; where is the evidence for this? no pilot testing

If there were garnering of evidence on the impact of policies, then ineffective policies could be weeded out

UP

Reg. Para Priv.

Salary/month, 2008(today after 6th Pay Comm.)

12,017(27,000)

3,000(3,500)

940(1400)

Absence rate 24.6 12.0 17.4

% time teaching 75.3 83.3 89.0

SchoolTELLS survey (2008)

Higher resources, lower effort

Structure of accountability matters more than resources

What kind of evidence?

What kind of evidence is useful / acceptable?

Distinguishing correlation and causation

The importance of methodology

Using the force of the federal chequebook to nudge researchers to use robust methods capable of yielding causal inferences.

Interest in evidence of impact Field of medicine long interested in evidence of impact

Recent upsurge of interest in impact evaluation in many fields nutrition, labour, governance, rural development, education, poverty

‘International Initiative for Impact Evaluation’ (3ie) established 2008

Network of Networks on Impact Evaluation (NONIE), comprised of: the OECD/DAC Evaluation Network (DACEN), the UN Evaluation Group (UNEG), the Evaluation Cooperation Group (ECG), and the International Organization for Cooperation in Evaluation (IOCE)

International Development Evaluations Association (IDEAS)

Development IMpact Evaluation (DIME)

Spanish Impact Evaluation Fund (SIEF)

What is impact evaluation?

Measuring outcomes (eg learning) is difficult

But measuring outcomes is NOT impact evaluation

measurement of net effects of a program on the outcomes of interest (e.g. on learning)

In IE, it is important to appreciate the difference between correlation and causation

Correlation vs Causation

As ice cream sales increase, number of drowning deaths increases sharply Therefore, ice cream causes drowning

Sleeping with one's shoes on is strongly correlated with waking up with a headache Therefore, sleeping with one's shoes on causes headache.

Taller people have higher earnings Therefore, higher height causes higher earnings

Salary and Height

Salary and Height?

Spurious Relationships This relationship is said to be “spurious” When we did the bivariate relationship, we said,

In reality, things look like this:

SalaryHeight

SalaryHeight

Gender

Think of

Sarva Shiksha Abhiyan

and its impact

Correlation vs Causation In achievement equation with

class-size, the coeff on class-size var is -0.46, i.e. an increase in class-size by 10 is associated with a 4.6 point lower ach mark

Can we say from this that: class-size is negatively related

to achievement? reducing class-size will lead

student achievement to rise? what is the diff between these?

what else determines ach? where is that included in the

way the reg. equation is written

Class size graph

y = -0.4635x + 76.437

01020304050607080

0 20 40 60 80

class size

mar

k

Quantitative impact evaluation methods

OLS production function (not causal relation) Instrumental Variables Panel data Randomised experiment Quasi-experimental

Propensity score matching method Each method has strengths & drawbacks RCT, PSM, Panel, IV

Evidence on impact of policy interventions

Performance related pay Duflo and Hanna, RCT Muralidharan/Sundararaman, RCT

Contract teachers Atherton & Kingdon, panel data Muralidharan et al, RCT Goyal & Pandey, OLS, school FE

DPEP Jalan and Glinskaya, PSM Schmid, IV

Private schools Desai et al, panel French/Kingdon, panel

Mid Day Meal Afridi, 2010, panel

Union membership Kingdon Teal, panel

Greater emphasis on evidence

NCERT evaluated impact of programs in 4 states, under TCF

Data becoming available – EI, ASER, DISE, SEMIS, NAS

What’s imp is the quality of the data / studies, i.e. robustness of methods/ designs; degree to which they tease out causal effect

Long term investment needed in capacity dev

MHRD / JRM approval for estb of National Assessment & Evaluation Centre

Some important considerations

Even evidence showing reliably what works is insufficient

Policy makers need to: seek, read such evidence discern good/bad evidence act on evidence, i.e. make evidence-based policy

Freedom from political interference

Independence from funding body, hence PPP better

Good evidence does not always permit good decisions

Even when we have good evidence a policy has capacity to substantially improve outcomes, there can be powerful political economy barriers to the implementation of policies.

E.g. Duflo and Hanna (Rajasthan) say : “Although this study suggests that a system of automatic monitoring with enforcement by physically remote agents who are prepared to enforce the rules is technically feasible and indeed provides better incentives for teachers, a later effort to introduce this system with higher-skilled, higher-status, and more politically powerful health-care workers ran into strong political obstacles (Banerjee et al. 2007b).” Another e.g. para T in UP

We need to understand the political economy constraints, and how can they be eased

Thanks

TASK WITH DIFFICULT WORDS  

Bihar govt. schools

UP govt. schools

Bihar/

ALLUP

Reg.Para

05Para

06 Reg. ParaPrivat

e All

Word meaning    

Not attempted 8.8 10.8 10.2 5.0 8.3 12.5 9.4

Wrong meaning 35.5 33.4 35.4 27.2 29.3 32.0 32.2

Partial meaning 12.4 14.8 17.3 16.9 13.5 15.1 14.9

Full meaning 42.4 41.0 37.1 50.8 48.9 40.4 43.4

Four difficult words are given. Please write their meaning using simple words

VOCABULARY TASKS : DO TEACHERS KNOW WORD MEANINGS ?

SUMMARIZING TEXTS : EXAMPLES FROM TEACHERS

TASK WITH PASSAGE

Bihar govt. schools

UP govt. schools

Bihar/

ALLUP

Reg.Para

05Para

06 Reg. Para Private All

Gave meaningful summary?

Not attempted 3.5 2.4 5.6 3.3 2.6 8.6 4.4

Irrelevant/wrong 25.4 27.0 40.5 28.6 40.2 37.9 33.0

Partially meaningful 25.4 15.9 22.5 9.9 16.2 16.4 17.9

Fully meaningful 45.6 54.8 31.5 58.2 41.0 37.1 44.7

SUMMARIZING TEXTS : CAN TEACHERS SUMMARIZE ?

TASK WITH PASSAGE  

Bihar govt. schools

UP govt. schools

Bihar/

ALLUP

Reg.Para

05Para

06 Reg. Para Private All

Summary easy to understand              

Easy to Understand 71.2 69.6 78.6 79.3 81.3 73.8 75.0

Are there any spelling errors?              

No spelling error 32.0 29.1 28.2 54.7 48.2 46.7 39.6

1-2 Spelling errors 44.3 38.5 25.6 30.2 35.5 34.4 35.3

>=3 spelling errors 23.7 32.5 46.2 15.1 16.4 18.9 25.1

SUMMARIZING TEXTS : CAN TEACHER SUMMARIZE

USING SIMPLE LANGUAGE

Of those writing meaningful summary

  Bihar UP All

TASK: PERCENTAGE WORD PROBLEM

Reg. Para 05

Para 06

Priv. Reg. Para Priv.

Not attempted 14.4 12.0 26.4 37.0 16.7 23.5 28.6 20.6

Incomplete 32.7 48.8 46.2 25.9 40.0 40.0 54.6 42.6

Wrong steps 5.8 6.4 5.5 11.1 10.0 3.5 1.3 5.7

Correct steps wrong answer 3.9 6.4 3.3 3.7 4.4 7.0 1.3 4.6

Correct answer no steps 0.0 1.6 3.3 0.0 1.1 4.4 2.6 2.1

Solved correctly

43.3 24.8 15.4 22.2 27.8 21.7 11.7 24.5

TASK 4 : PERCENTAGE PROBLEM : Findings

Bihar UP All

AREA PROBLEMTASK Reg.

Para05

Para06-07 Priv. Reg. Para Priv.

No attempt 27.9 28.8 38.5 51.9 30.0 48.7 41.6 36.6

Incomplete 19.2 25.6 26.4 7.4 18.9 19.1 26.0 21.8

Wrong steps & A 5.8 4.0 1.1 3.7 7.8 3.5 2.6 4.1

Correct steps, wrong A 3.9 3.2 8.8 0.0 4.4 1.7 5.2 4.1

Only correct A, no steps 4.8 5.6 3.3 0.0 8.9 4.4 9.1 5.5

Solved correctly 38.5 32.8 22.0 37.0 30.0 22.6 15.6 27.9

TASK 5 : ARITHMETIC : AREA PROBLEM

  BIHAR UP

  Fully agree

Parti ally

agree

Somewhat agree

Dis agree

Fully agree

Parti ally

agree

Somewhat agree

Dis agree

Govt. school teachers

24.5 11.0 46.8 17.7 15.2 18.3 43.1 22.3

Private school teachers

16.7 12.5 45.8 25.0 16.9 18.5 36.9 27.7

% teachers who agree with the statement“Sometime I have difficulties in addressing mathematical queries and

problems of my students”

80% primary school teachers have difficulties in teaching maths

Only about 20% of govt. school teachers believe they don’t face problems. About 80% admit to have difficulties sometimes. This suggests possible interest in in-service training to upgrade maths skills