Excuse me, do you speak English? An International Evaluation using ESLC

Excuse me, do you speak English?An International Evaluation using ESLC

Ester Nunez de Miguel

CEMFI

March 23rd, 2015

Ester Nunez de Miguel (CEMFI) Excuse me, do you speak English? March 23rd, 2015 1 / 29

Outline

1 Introduction

2 The European Survey of Language Competences

3 The Education Production Function

4 English Class Size

5 Conclusion


1 Introduction




5 Conclusion


Introduction

Economists are interested in educational topics since we considerEducation is an Investment

I of society in today youngsI in terms of the youngs’ personal effort

This Investment ends up with individuals with greater and morediverse skills, whose final reward are better employment opportunitiesonce they finish their studies.

Nowadays, English knowledge has become a very valuable skill in thelabor market. International trade and globalization have made Englishan essential tool for worldwide communication.

Due to its importance, countries allocate a great amount of publicexpenditure in education, making special emphasis on English as aforeign language.


Differences in the educational systems on the English onset and theimportance given to foreign languages varies greatly across countries,and this creates huge differences in World English levels.

From a quantitative point of view, economists search which teachingmethods are more effective in terms of results.

In terms of educational policy, economists search which policychanges can improve educative results with a given amount ofavailable resources.

Until now, the lack of comparative English test and teaching dataprevented an exhaustive applied analysis to be done.


European Survey of Language Competences (ESLC) 2011 its the firstdatabase of its kind, and allows to evaluate

Which are the main determinants of English proficiency at the end ofcompulsory education?

Explore which factors effect differ from previous research in otherskills previously evaluated.

Use Instrumental Variables techniques to address the identification ofthe effect of English Class size on outcomes.

“If custom and law define what is educationally allowable within a nation,educational systems beyond one’s national boundaries suggest what iseducationally possible.”

W. Foshay (1962), first pilot study of international student achievement.


1 Introduction




5 Conclusion


The ESLC

The European Survey on Language Competences (ESLC) 2011:

14 countries participated and were evaluated in their 2 first targetlanguages (among English, French, German, Spanish and Italian):

I Tested Educational level: Last year of lower secondary education.Ages: 14-16 years old

English was the first target language for 13 educational systems.I The test evaluates 3 skills: Reading, Listening and Writing

I Each student was evaluated in 2 skills and completed a studentquestionnaire (SQ): 23,358 students

I Teacher Questionnaire (TQ)

I Principal Questionnaire (PQ): 1047 schools

I National Information


The ESLC

The sample was designed to satisfy the technical requirements forinternational testing, ex: PISA, TIMSS.

A routine test previous to the skill evaluation assigned each studentto a test level: A1-A2, A2-B1 and B1-B2.

Each student did 5 tasks per skill: each task measures only within alimited range; results are not comparable across students.

Results of the test are reported in terms of the Common EuropeanFramework of Reference (CEFR) which has 6 levels of functionalcompetence: A1 (lowest) - C2 (highest).

ESLC Level CEFR Level DefinitionIndependent User Advanced B2 Can express herself clearly and effectively

Beginner B1 Can deal with straightforward familiar matters

Basic User Advanced A2 Can use simple language to communicate on everyday topicsBeginner A1 Can use very simple language

Beginner Pre-A1 Have not achieved the A1 level


Mean Test Results by Country


1 Introduction




5 Conclusion


Education Production Function

Economic literature has made use of international test of educationalachievement to analyze the determinants and impacts on cognitive skills.

Our purpose has been the estimation of an education productionfunction using ESLC:

Sisc = β0 + β1Fi + β2Ris + β3Ic + β4Ai + εisc

Sisc Listening, Reading and Writing.

Fi captures student and family background characteristics

Ris school resources, I institutional features

Ai individual ability

The cross-sectional nature of this estimation allows only for a descriptiveinterpretation of which are the determinants of English performance


Specification

Student and Family background characteristicsI Studenti : age, gender, immigrant, repetition;I Familyi : Father and Mother English level, education, books at home,

profession, English use at home, early onset at home;

School ResourcesI Studentis : number of years of English at school, early onset at school,

English lesson time per week, English class size, ancient languages,number of foreign languages;

Institutional InformationI School : External Exit ExamsI Country : GDP 2010, Educational Expenditure 2010, TV in English is

subtitled, Multilingual region

Family Language ControlsI Germanic: Netherlands, SwedenI Romance: French Belgium, France, Malta, Portugal, SpainI Slavic: Bulgaria, Croatia, Poland, Slovenia


Available Data

Original sample size: 23,358 students

Following the Main SampleI Remove 4 schools where no SQ was reportedI 5,228 observations are lost due to missing values in the SQI 40% of the sample is lost due to missing values in PQ and TQ

Final available data: 14,964 students.I S Listening observations: 11,116I S Reading observations: 11,225I S Writing observations: 10,951

Estimation Method

Least Squares Regression weighted by students sampling probability⇒ Population Effects


Environmental Factors

Mean Mean MeanSTUDENT BACKGROUND Listening Reading Writing

Age -0.14*** -0.09*** -0.18**(0.028) (0.03) (0.085)

Female 0.02 0.01 0.48***(0.03) (0.03) (0.08)

Repetition -0.18*** -0.35*** -1.26***(0.058) (0.056) (0.13)

Immigrant 0.23*** 0.10 0.27(0.08) (0.08) (0.21)

ESCS index 0.13*** 0.15*** 0.26***(0.03) (0.03) (0.10)

Observations 11,116 11,225 10,951R-squared 0.502 0.442 0.439

Clustered robust standard errors by schoolin parentheses, *** p<0.01, ** p<0.05, * p<0.1

In line with PIRLS studies, Girls perform better in writing skills.

Immigrant is a positive and significant determinant of Listening skills.

Repetition effect is negative and significant for all skills.

ESCS index is a positive and significant predictor of studentperformance.


Environmental FactorsMean Mean Mean

FAMILY BACKGROUND Listening Reading Writing

Father University 0.085** 0.09* 0.28**(0.04) (0.047) (0.121)

Mother University 0.113*** 0.11** 0.35***(0.043) (0.05) (0.106)

White Collar Father 0.02 0.04 -0.12(0.039) (0.047) (0.117)

White Collar Mother 0.18*** 0.21*** 0.57***(0.049) (0.049) (0.14)

Blue Collar Father -0.09** -0.12*** -0.42***(0.039) (0.04) (0.13)

Pink Collar Mother 0.10*** 0.08* 0.48***(0.036) (0.046) (0.129)

Father English Well 0.16*** 0.15*** 0.48***(0.035) (0.035) (0.098)

Mother English Well 0.04 0.05 0.13(0.032) (0.04) (0.099)

English use at home 0.41*** 0.35*** 0.98***(0.04) (0.05) (0.13)

Early Onset at home 0.23*** 0.23*** 0.41**(0.074) (0.08) (0.18)

Clustered robust standard errors by schoolin parentheses, *** p<0.01, ** p<0.05, * p<0.1


Environmental Factors

Family Background

Mother and Father University have a positive and significant impact.

Blue Collar and White Collar have a negative and a positive effectrespectively in all the skills.

I The fact that only one of the parents affects the kid could be driven byrelative impact or by collinearity due to assortative mating.Correlation between parents education expressed in years: 62,4%.

Father’s English knowledge is a positive and significant determinant.

Early onset at home: English knowledge at home before 5 years old.


Environmental FactorsMean Mean Mean

NATIONAL INFORMATION Listening Reading Writing

GDP ppa 2010 per capita -0.07*** -0.06*** -0.18***(0.02) (0.02) (0.05)

Educational Expenditure 2010 0.16*** 0.15** 0.62***(0.06) (0.06) (0.14)

Un-dubbed English TV 0.16 0.20* 0.47*(0.10) (0.12) (0.27)

Population 2011 -0.11*** -0.06*** -0.20***(0.02) (0.017) (0.04)

Multilingual Region 0.28** 0.08 0.65(0.13) (0.16) (0.42)

External Exit Exams -0.11 -0.35*** -0.67***(0.08) (0.09) (0.2)

*** p<0.01, ** p<0.05, * p<0.1

Educational Expenditure in 2010 at secondary is positive and signif.

Un-dubbed English TV is positive and significant.

Multilingual regions perform positive and significantly in Listening.

External Exit Exams are negative and signif: teaching-to-the-test?


What happens in the School?

Which educational policies can significantly change the results in Englishperformance at the end of compulsory education?

Mean Mean MeanSCHOOL INFORMATION Listening Reading Writing

English Hours per Week 0.08*** 0.1*** 0.27***(0.026) (0.031) (0.067)

Years English School >5 years 0.03*** 0.05*** 0.08***(0.006) (0.006) (0.02)

Early Onset at School <5 years 0.17*** 0.17*** 0.51***(0.06) (0.06) (0.13)

Number of Foreign Languages 0.11*** 0.11*** 0.21***(0.024) (0.026) (0.06)

Ancient Languages 0.07 0.13** 0.39***(0.054) (0.058) (0.14)

English Class Size logarithms 0.16*** 0.19*** 0.75***(0.05) (0.057) (0.17)

*** p<0.01, ** p<0.05, * p<0.1

The English class size effect is positive and significant: biggerclasses perform better on average.


1 Introduction




5 Conclusion


English Class Size

English Class size is a positive and significant determinant in the 3 skillsanalyzed ⇒ Bigger classes perform better on average than smaller ones.

Class sizes depend on choices by politicians, administrators andparents which may be related to the level of performance achieved ⇒A more accurate estimation method is needed to get the effect ofEnglish Class Sizes on Results.

Economics is a social science, therefore we can not experiment withstudents to estimate the effect of class size changes.

Regulation induces changes in the class size ”treatment”

Angrist and Lavy (1999) Maimonides’ rule exploiting maximumclass size regulation is the best known example.

Maximum Class size induces nonlinear and non monotonic relationshipbetween grade enrollment and class size ⇒ Creates exogenous variation


English Class SizeA simple example

Country with Regulation Maximum Class Size (25)

Treatment: English Class SizeNumber of Number of

School Cohort Size Classes Students per Class

A 25 1 25B 26 2 13C 50 2 25D 51 3 17E 75 3 25

School A has 25 students in the course ⇒ 1 class ⇒Treatment: English Class Size = 25

School B has 26 students in the course ⇒ 2 classes ⇒Treatment: English Class Size = 13

English Class Size Treatment differs only due to the regulation that affectsmaximum class size (25 in the example)


English Class Size

According to Eurydice, 9/13 countries in ESLC sample havemaximum class size regulations.

Principals of each school report the number of English students pergrade (Cohort size).

The dynamics of the previous example can be modeled to create thePredicted Class Size function.

Predicted Class Size can be used as an instrument of the actualclass size, therefore, it allows to create exogenous variation and infercausality from class size.


Example: Portugal


English Class Size

Mean Listening Mean Reading Mean WritingEnglish Class Size logs OLS IV OLS IV OLS IV

Bulgaria 1.01** 0.41 0.59** -3.57* 0.8 -1.65Estonia 0.46*** 0.531 0.164 0.565 0.329 1.258Greece 0.10 0.057 0.17 -0.005 0.67*** 1.3Spain 0.16 0.23 0.31*** 1.979* 0.47 1.35France 0.29 -0.25 0.27 1.06** 1.23* 4.27**Portugal 0.16 0.06 -0.14 0.16 0.002 -0.72Slovenia 0.34** 4.81 0.077 0.87 -0.16 -0.82Sweden 0.42*** 0.05 0.70*** 2.08** 1.39*** 4.46**

Previous Estimation [13] 0.16*** 0.19*** 0.75***Countries with Regulation [8] 0.19** -0.029 0.28*** 1.01*** 0.65** 1.84**

*** p<0.01, ** p<0.05, * p<0.1

Method 1 (OLS) estimates by country present the same positivepattern in English Class Size effect previously shown.

Method 2 (IV) coefficients are no longer statistically significant for allbut for Reading and Writing in Spain, France and Sweden.


English Class Size

What previous results have shown, after using IV estimation of ClassSize effects, was that IV coefficient were negative or no longersignificant.

The Writing skill IV Class Size coefficient results in Spain, France andSweden are positive and even greater than the OLS coefficients.

Since English is a multi-skill ability, teachers that face bigger classes canmake special emphasis on one skill versus some other. If the teacher has abigger class, he/she could dedicate greater part of the class to grammarand exercises correction rather than speaking or listening activities. Thisfocus on grammar can make bigger classes more effective in EnglishReading and Writing skills.


1 Introduction




5 Conclusion


Conclusion

During this study we wanted to explore the main determinants ofEnglish outcomes performance using an European sample (ESLC).

Since English class size was a positive and significant determinant ofall skills we explore the causality using IV Maimonides’ rule.

The positive and significant result obtained by IV in Reading andWriting skill in 3 countries has a clear policy implication: The factthat bigger classes performed on average better in Writing indicatesthat bigger classes emphasize more grammar and writing activities ata cost of reducing the relative time allocated to other English skills.


Thank you very much!

Questions? Suggestions?
