How much are we learning? Advances in school evaluation in New York and São Paulo

How much are we learning?

Advances in school evaluation in New York and São Paulo

May 19, 2010

Jesse Margolis

Instituto Braudel

2

• General observations about São Paulo schools

– Shared characteristics with New York

– Notable differences

• Brief presentation of New York City’s accountability system

• School evaluation in São Paulo

– Best practices

– Opportunities for improvement

– Recommendations


3

Schools in São Paulo share challenges with New York• Both states enroll a large number of public school students…

– New York State has 2.7 million students enrolled in 4.6K public schools– São Paulo State has 7.8 million students enrolled in 11.8K public schools (approx.

10 Federal, 5.7K State and 6.1K Municipal)

• …and contain big cities with a substantial share of public school enrollment– New York City has 1.0 million students enrolled in 1.5K public schools– The city of São Paulo has 2.0 million students enrolled in 1.6K public schools

(approx. 2 Federal, 1.1K State and 500 Municipal)

• In both cities, a significant share of school-age children are enrolled in private schools…– In New York City, 19% of all students are enrolled in private schools, compared with

10% in the rest of the state and 10% in the entire United States– In the city of São Paulo, 18% of all students are enrolled in private schools,

compared with 14% in the rest of the state and 12% in all of Brazil

• …leaving the public school system primarily for low-income families– In New York City, 66% of public school students are eligible for free lunch due to low

family income– In São Paulo, 68% of public school parents in the State system report an income of

R$1,275 (about US$700) or less per month

Source: NYSED, Censo Escolar 2008, SARESP Cuestionário 2008; Note: excludes “eucação infantil” for São Paulo and PK for NYC.

4






– Best practices


– Recommendations


5

• State: regulation and oversight• Establish standards• Certify teachers• Evaluate schools and districts• Define physical boundaries and

governance of school districts• Redistribute funding between

wealthier and poorer school districts

• City: run schools• Build buildings• Hire principals, teachers and other

staff & set salaries• Develop an admissions system for

students• Decide on details of curriculum &

professional development• Pay staff and other expenses

Responsibilities for school governance are divided differently in São Paulo and New York

0%

20%

40%

60%

80%

100%

ElementarySchols

MiddleSchools

High Schools

Per

cent

age

of S

tude

nts

São Paulo: State and municipal governments both regulate and run schools

StateSchools

MunicipalSchools

New York: State regulates school system and city runs schools

2.5MM 2.5MM 1.5MM

Source: Censo Escolar 2008

6

0

50,000

100,000

150,000

200,000

Accoun

tants

Registered

nurses

Sales(m

anufac

turin

g)

ES,M

S&HS

Teache

rs

Policeofficers

Officesu

perviso

rsCa

rpen

ters

Exec

secretary

Truc

kdrivers

Mainten

ance

workers

Accoun

tingclerks

Custom

erse

rvicereps

Secretaries

Nursingaide

s

Secu

ritygu

ards

Janitors

andcle

aners

Offic

ecle

rks,

gene

ral

Retailsalesp

ersons

Teac

hera

ssist

ants

Waiters

Homehe

alth

aide

sStoc

kclerks

Cash

iers

Chief executivesPhysiciansLawyers

FinancialmanagersOperationsmgr.

Teachers in New York State earn above the median salary, but below most other professions requiring a college degree

US$ 65.899 / year

Ave

rage

Sal

ary

($U

S /

year

)

Note: includes categories 25-2031, 25-2022 and 25-2021 for high, middle, and elementary school teachers. Does not include kindergarten, pre-school, or special education teachersSource: Occupational Employment Statistics (OES) Survey, May 2008. Bureau of Labor Statistics and Department of Labor

7

0

2,000

4,000

6,000

8,000Prof

(ens

inosu

perio

r)Ad

voga

dos

Gerentes

operaç

ões

Gerentes

deap

oio

Secretáriasex

ecutivas

Enferm

eiros

Professo

resEF

&EM

Cond

utores

(distrib)

Policiais

Aux.

enferm

agem

Escritu

rário

sem

geral

Vend

edores

emlojas

Alve

naria

Guarda

sevigias

Garçon

seba

rmen

Embe

leza

men

to

Recepc

ionistas

Telemarke

ting

Cozin

heiro

sTrab

alha

doresag

rícolas

Manu

tenç

ãode

edifício

s

Costurade

roup

as

Serviço

sdo

méstic

os

Aten

d.de

crec

he/id

ososMédicos

DiretoresgeraisEngenheiroscivis

Dirigentesde empresas

Reps

comerciais

Source: PNAD 2008

Ave

rage

Sal

ary

($R

/ m

onth

)In São Paulo State, the same is true, though salaries across the board are lower

0

2,000

4,000

6,000

8,000

10,000Mé

dico

s

Enge

nheiroscivis

Cirurgiões-den

tistas

Enge

nheirosmecân

icos

Prof

(ens

inosu

perio

r)

Pesq

.Econô

mica

Contad

ores

Journa

listas

Advo

gado

s

Diretoresde

prod

ução

Psicólog

os

Enferm

eiros

Farm

acêu

ticos

Fisioterap

eutaseafins

Professo

resEF

&EM

8

Among professions in São Paulo with a high share of university degrees, elementary and secondary teachers are the largest and lowest paid group

Note: includes only those professions where 80% or more of respondents said that their last level of education studied was “Superior – graduação” or “Mestrado ou doutorado”“Source: PNAD 2008

Ave

rage

Sal

ary

($R

/ m

onth

)

9

Full-time teachers in São Paulo spend a larger share of their contractual time teaching students

0

5

10

15

20

25

30

35

40

45

NYC (High Poverty) NYC (Low Poverty) São Paulo

Teac

hing

per

iods

(50

min

utes

)

Teaching(22 periods)

Preparation(8 periods)

Professional Activity (5 per.)

Lunch/Transition(6 periods)

41 periods(34hr 10min / week)


Preparation(5 periods)

Professional Activity (5 per.)

Lunch/Transition(6 periods)



HTPC* (3 periods)

Prep. (4 periods)


In or out of school (not teaching)In school (not teaching)In school (teaching)

* HTPC is shared planning and professional development timeSource: NYC DOE teachers contract (2007-2009); DECRETO Nº 55.078, DE 25 DE NOVEMBRO DE 2009

10

46% of public school teachers in the São Paulo State system report working in two or more schools

0%10%20%30%40%50%60%70%80%90%

100%

Elementary SchoolTeachers (4th

Grade)

Middle and HighSchool

(PortugueseLanguage)

Middle and HighSchool (Math and

Physics)

Middle and HighSchool (Chemistry

and Biology)

Respondents: 7,102 15,874 17,542 11,763

Per

cent

age

of T

each

ers

Source: Qúestionario SARESP 2008

One School

Two Schools

Three or more

11

Across all grade levels, average class sizes in São Paulo are larger than in New York

Note: Sao Paulo class size is for Regular Education in Urban, State Schools. New York City class size is for General Education and Gifted & TalentedSource: 2008 Censo Escolar for São Paulo; 2009/10 Class Size Report for New York City

31 32 31 3235 35 35 36

3736 35

22 2325 25

26 27 28 27 27 27 27

0

5

10

15

20

25

30

35

40

2 3 4 5 6 7 8 9 10 11 12

Ave

rage

Cla

ss S

ize

São Paulo State (Avg. = 34.7)(Urban schools)

Grade Level

New York City (Avg. = 26.1)

12

Teacher absenteeism is two and a half times greater in São Paulo State than in New York City

15%

6%

0%

2%

4%

6%

8%

10%

12%

14%

16%

São Paulo(2008)

New York City(2006)

Per

cent

of S

choo

l Day

s

Average Teacher Absence Rate

Source: São Paulo figure is based on the 2008 school year from Veja interview with Paulo Renato Souza on 10/24/09. New York City figure is from Chicago Tribune article “Teachers miss days; poor kids miss out; Educators at some struggling schools take most time off, analysis shows, September 25, 2006.Teacher quote from “Paralisação dos professores em SP tem baixa adesão”, Estado de São Paulo, March 9, 2010

Recent Policies in São Paulo Have Focused on Teacher Attendance

• Teacher Bonus: beginning in 2008, teacher performance bonuses of up to 22% of salary are proportional to their attendance (67% attendance minimum)

• Teacher Raises: beginning in 2010, teachers must meet attendance minimums to be eligible for raises of 25%

• “I don’t even think of going on strike, because if I am absent, I lose the one chance of earning a real raise”

• São Paulo Public School Teacher, March 9, 2010.

13

More high school students in São Paulo go to school at night

43.6%

5.7% 5.5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

São Paulo State Schools São Paulo Private Schools New York City PublicSchools

Enrollment: 981,767 132,593 331,437

Per

cent

of H

igh

Sch

ool S

tude

nts

Source: Censo Escolar 2009 (Sinopse Table 2.16); DOE ATS System as of 4/02/10

14

• Students in São Paulo go to school for fewer hours

– The school year is longer (200 days in São Paulo vs. 185 days in New York)

– The school day for the student is shorter (4 hours to 5 hours and 20 minutes in São Paulo vs. 6 to 7 hours in New York)

Other observations about students

• Students in São Paulo take more courses at the same time

– São Paulo: students are normally enrolled in 10 to 13 courses

– New York: students are normally enrolled in 5 to 7 courses

15

• Students in São Paulo

V go to school for fewer hours each week

V and study more subjects at the same time

• Teachers in São Paulo

V teach more hours each week

V have larger class sizes

V and are more likely to teach in multiple schools

Middle and High School Teachers in São Paulo have a substantially higher student load than in New York

Teachers have a much higher student load

Teachers need to know the names and track the progress of many more students

16






– Best practices


– Recommendations


17

New York: Accountability Framework

WH

AT

EVALUATE ENFORCE CONSEQUENCES ENABLE

HO

W

Progress ReportsGrades based on student outcomes

Learning Environment SurveyParent, teacher, and student surveys

Quality ReviewsScores based on performance management criteria

DemandEnrollment demand

Rewards Monetary bonuses for principals and a pilot program for teachers

ConsequencesImmediate restructuring of chronically failing schools. Target setting for other D/F schools; if no improvement, leadership change and/or closure

EmpowermentPrincipals have more autonomy over budget, hiring and professional development

Periodic Assessments Diagnose and track progress

Achievement Reporting and Innovation System (ARIS)Fully integrated knowledge and data management

Children First IntensiveHands-on data training through Inquiry Teams

18

The Progress Report was designed with several guiding principles in mind:

• Outcomes, not inputs: Focus the conversation on student outcomes rather than inputs

• Performance and Progress: Measure performance and progress of individual students

• City and Peer Comparison: Compare schools to similar schools as well as a citywide standard

• Simple, Valid, Verifiable: Ensure that schools can verify and re-create metrics, so schools understand how they are measured and how they can improve their performance

Progress Report guiding principles

19

Progress Report measures

School Environment

15 points

Student Performance

25 points

Student Progress

60 points

Additional Credit

Up to 15 points

• Learning Environment Survey results

• Attendance

• Student test scores in ELA and Math (median proficiency and % Level 3/4)

• Individual student progress on ELA and Math test scores

• Graduation rates (4-year and 6-year)

• More points for diplomas indicating college-readiness

• Exemplary progress on test scores with high need students

• Exemplary progress in credit gains with high need students

• Learning Environment Survey results

• Attendance

• Credit accumulation

• Regents completion and pass rates

Elementary, Middle, and K-8 Schools

High Schools

20






– Best practices


– Recommendations


21

SARESP is a good instrument to measure student learning

• Performance Scale: Uses Item Response Theory (IRT) to assign each student a score for each subject, based on a scale from 0 to 500, with the majority of students between 125 and 400

• Proficiency Levels: In each subject, the scale is divided into four levels:

• Advanced

• Adequate

• Basic

• Below Basic

22

IDESP is a simple way to combine two different objectives: improve student performance and improve promotion rate

• IDESP is based on two indicators:

• Performance Indicator: Based on the proficiency levels of students on standardized Portuguese Language and Math tests

• Promotion Indicator: Grade promotion rate of the students

IDESP = Performance Indicator X Promotion

Indicator

23

IDESP has a well thought out system of long and short-term targets

Source: IDESP Technical Note, Secretaria Estuadual de Educação do Estado de São Paulo

• Long term targets (2030): The São Paulo State Education Department (SEE-SP) established long-term goals based on simulating IDESP results for developed countries in the OECD

• School-specific targets: SEE-SP established intermediate targets for each school, based on the distance between its starting point and the long-term goal

24

IDESP is relatively simple to calculate, though the State’s current explanation could be improved

The Performance Indicator (ID) isbased on the proficiency levels• Av = percentage of students at the advanced level • Ad = percentage of students at the adequate level• B = percentage of students at the basic level• AB = percentage of students below basic (abaixo do básico)

A. Two-step formula described in the Nota Tecnica de IDESP

Step 1: Calculate “Desfasagem” Step 2: Calculate Performance Indicator

B. Equivalent one step formula

IDjs = (10 x Av) + (6,66 x Ad) + (3,33 x B) + (0 x AB)

Note: an alternative formula for “B” would be IDjs = (10 x Av) + (20/3 x Ad) + (10/3 x B) + (0 x AB)

25

Positive: IDESP is given strong teeth by the bonus program

22.3%

4.6%

0%

5%

10%

15%

20%

25%

São Paulo State New York City

Per

cent

of A

nnua

l Sal

ary

Maximum Bonus as a Percent of Average Annual Teacher Salary

Fonte: New York City bonus is $3,000 per teacher provided to schools in a 200-school pilot project that meet Progress Report targets. São Paulo bonusis up to 2.9 additional monthly salaries for teachers whose schools improve their IDESP scoreSource: teacher salaries based on PNAD 2008 for São Paulo and Bureau of Labor Statistics May 2008 Occupational Employment Statistics survey

26






– Best practices


– Recommendations


27

Change in IDESP from 2007 to 2008

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

IDESP 2007

IDES

P 20

08

Note: each school is represented by a separate point.Source: Boletim IDESP

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

IDESP 2008

IDES

P 20

09

28

Change in IDESP from 2008 to 2009

Recommendation: Interview school personnel to identify best practices and review testing micro-data to identify potential weaknesses in the reliability of test score and passing rate indicators.

Note: each school is represented by a separate point.Source: Boletim IDESP

29

Recommendation: incorporate percentage of students taking the test into IDESP, potentially by multiplying the test taking percentage by the IDESP score

There is no incentive for weaker students to take the test

Source: Boletim SARESP 2008; Censo Escolar 2008; Author interviews.

93%85%

78%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

5th Grade 9th Grade 12th Grade

Average Test Taking Percentage (SARESP 2008)

Test Taking Percentage by School (SARESP 2008, Grade 9)

• “There were many schools that only sent the good students, and if you only send the good students, your IDESP is better. The great failure of IDESP is that it does not consider the number of students who take the test”

-- High School Principal, São Paulo State

0

50

100

150

200

250

40% 50% 60% 70% 80% 90% 100%

30

“SARESP is not a tool for improving education...we don’t have access to the student test results”

- High School Teacher, São Paulo State

Individual student or item results are not provided to the schools, limiting SARESP’s use in improving education

Recommendation: Develop a plan to provide individual test results to schools and parents.

“Who are the students who are below basic; we don’t know What did they get wrong; we don’t know...How am I going to work with the students this year if I don’t know what they got wrong”

- High School Curriculum Director, São Paulo State

Source: author interviews

31Note: parent survey is given once per year to parents of all students who take the SARESP (5 th, 7th, 9th, 12th grades)Source: SARESP 2008 Questionário dos pais

The parent survey asks many questions that can paint a picture of a school’s learning environment

Parents: To what degree to you agree or disagree with the following statements:

Communication with parents• 2.1. I receive information from the school about my child’s progress.• 2.11. The school always holds parent meetings to inform me about my child.

Parent Involvement• 2.17. When there is a problem I am quickly called to the school.• 2.19. The school recognizes that parents’ opinions are important.

Student Safety• 2.3. My child is safe in school.• 2.16. My child feels safe in school.• 2.23. This school has many students with behavioral problems.

Opinion about teachers• 2.4. The school’s teachers respect the students.• 2.12. I believe the teachers are very capable.

Opinion about the school• 2.15. I would like my child to study in another school.• 2.24. If I could pay, my child would study in private school.

32Source: SARESP 2008 Questionário dos pais

48%40%

57%

30%

36%41%

38%

23%

17% 19%5%

48%

58%

32%

10%

0%10%20%30%40%50%60%70%80%90%

100%

2.1. I receiveinformation fromthe school about

my child’sprogress

2.19. Theschool

recognizes thatparents’

opinions areimportant

2.3. My child issafe in school

2.12. I believethe teachers are

very capable

2.15. I wouldlike my child to

study in anotherschool

Completely Agree

Partly Agree

Disagree

1,174,660 1,119,737 1,130,132 1,195,661 1,136,501

The survey could give valuable system-wide information for the State of São Paulo

Per

cent

age

of R

espo

nden

ts

33Source: SARESP 2008 Questionario dos pais

58%

48%

40%

57%

30%

77%

61%

69% 71%

9%

27% 29%

22%

37%

43%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

2.1. I receiveinformation fromthe school about

my child’sprogress

2.19. The schoolrecognizes that

parents’ opinionsare important

2.3. My child issafe in school

2.12. I believethe teachers are

very capable

2.15. I would likemy child to studyin another school

Per

cent

age

that

Com

plet

ely

Agr

ee

São

Pau

lo S

tate

Sch

ool A

Sch

ool B

Recommendation: publish results of parent, teacher, and student surveys, aggregating results to maintain confidentiality

The survey could provide parents, teachers, and students with an insider’s perspective into school quality

34

4.5

5.25.7

6.1

0

1

2

3

4

5

6

7

D C B A

N* = 2 11 138 886

NYC Simulation: On average, schools with better Progress Report letter grades had higher simulated IDESP scores in 2009

Sim

ulat

ed ID

ES

P (2

009)

Note: Includes Ensino Fundamental I (4th grade) and Ensino Fundamental II (8th grade) separately, so schools with both grades may appear twice in the chart.

IDESP Simulation: However, there was substantial variation around these trends

R = 0.20

0

20

40

60

80

100

120

140

2 3 4 5 6 7 8 9 10

IDESP Simulado (2009)

Res

ulta

do n

o B

olet

im d

e Pr

ogre

sso

(200

9)

Note: Includes Ensino Fundamental I (4th grade) and Ensino Fundamental II (8th grade) separately, so schools with both grades may appear twice in the chart.

Simulated IDESP Score (2009)

Pro

gres

s R

epor

t Ove

rall

Sco

re (2

009)

36

1. The two instruments measure different things

• NYC Progress Reports include parent survey as 10% of the grade

• IDESP includes student grade promotion rates

2. NYC Progress Reports include more students

• IDESP only includes the last year of elementary school and last year of middle school

• Progress Report includes all students that take the test: 3rd – 5th in elementary school and 6th – 8th in middle school

3. 75% of the NYC Progress Report is based on a comparison between similar schools; IDESP is calculated equally for all schools in the state.

4. The NYC Progress Report measures students’ individual progress; the change in IDESP compares the results of one group of students in a particular grade one year with a different group of students in the same grade the following year

Hypotheses to explain difference between IDESP and NYC Progress Reports

Recommendations: Consider “Additional Credit” for: • Individual student progress• Performance relative to similar schools

37

Recommendation: Continue developing tools to help principals, teachers, and other school professionals do their job better

• Excel tool that allows the school to simulate student test performance and grade promotion results and see the impact on target achievement and teacher bonus

IDESP Modelo

Escola:

Lingua Portuguesa 2009 Atual* 2010 Proj. 2009 Atual* 2010 Proj. 2009 Atual* 2010 Proj. EF1 EF2 EMAvançado (AV) 4% 4% 1% 1% 0% 0% IDESP 2009 2.93 2.08 0.79Adequado (AD) 21% 21% 11% 11% 17% 17% Meta 2010 3.09 2.21 0.88

Básico (B) 48% 48% 56% 56% 31% 31% Proj 2010 2.88 2.06 0.79Abaixo do Básico (AB) 26% 27% 32% 32% 52% 52%

Total 99% 100% 100% 100% 100% 100%

Indicador de Desempenho (LP) 3.4615 3.400 2.7451 2.700 2.1839 2.167

MatimáticasAvançado (AV) 2% 2% 1% 1% 0% 0%Adequado (AD) 14% 14% 4% 4% 2% 2%

Básico (B) 48% 48% 58% 58% 19% 19%Abaixo do Básico (AB) 36% 36% 37% 37% 79% 79%

Total 100% 100% 100% 100% 100% 100%

Indicador de Desempenho (Mat) 2.7677 2.733 2.3039 2.300 0.7471 0.767

Taixa de Promoção 94.0% 94.0% 82.4% 82.4% 54.0% 54.0%

IDESP 2.93 2.88 2.08 2.06 0.79 0.79

Meta para 2010 3.09 2.21 0.88

% de meta alcançada 0% 0% 2%

* No cálculo atual de 2009, os porcentagems em cada nível estão com uma grau de precisão maior de o que esta disponivél aqui Isso resulta que as sumas das porcentagems pode não igualar 100% por causa de "rounding"

Ensino Fundamental I Ensino Fundamental II Ensino Médio

2.93

2.08

0.79

3.09

2.21

0.88

2.88

2.06

0.79

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

EF1 EF2 EM

IDES

P

IDESP 2009Meta 2010Proj 2010

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I understand how the IDESP isrelated to the teacher bonus

Per

cent

of R

espo

nden

ts

n = 37

Completely agree

Agree

Disagree

Completely disagree

Survey with Teacher Coordinators (April, 2010) IDESP Modeler

38






– Best practices


– Recommendations


39

• Short-term

• Simplify the explanation of IDESP and implement active training effort so that school staff understand the calculation

• Further study why certain schools had large jumps in IDESP to identify best practices or test-security weaknesses

• Incorporate the percentage of students taking the test into the calculation of IDESP

• Provide school-wide parent, teacher and student survey results to the public, aggregating results at the school level to maintain confidentiality

• Long-term

• Develop a plan to provide individual student test results to schools and parents

• Consider incorporating individual student progress and performance relative to similar schools into the measure

• Continue developing tools to help school principals, teachers and other staff do their job better

Recommendations

40

Thank you!

Jesse Margolis [email protected]

Documents

How much are we learning? Advances in school evaluation in New York and São Paulo