Slide 1

Education research: Issues, sources and tools Prepared for the DLI Workshop Kingston, Ontario April 7, 2004 Maryanne Webber & Franois Nault Culture, Tourism and Centre for Education Statistics

Slide 2

Centre for Education Statistics Mandate: deliver comprehensive program of national education statistics for policy development, educators and the public Overarching conceptual framework Close to 30 institutional surveys and household surveys New directions in dissemination

Slide 3

Conceptual framework: work in progress Labour Market Organizations in the private and public sectors Individuals Skills Market Families Demand Supply Transition between markets E-S PSE Adult ed.

Slide 4

Conceptual framework: work in progress Labour Market Organizations in the private and public sectors Individuals Skills Market Families Demand Supply Transition between markets E-S PSE Adult ed. Problematiques: skills market How effective are learning systems in ensuring people have the necessary skills? How to increase high school graduation and access and persistence in pse? What are the most cost effective ways to achieve learning and skill development? Do all segments of society have equal opportunity? What can be done for those left behind? What is the role of ICT in education? Problematiques: labour market What is the level of skill among Canadians? How to facilitate the transition? How does skill influence life outcomes? What are the factors influencing skill gain and loss?

Slide 5

Types of surveys Skills supply: Institutional surveys of enrolment, graduates, teaching staff and education finances; Information and Communication Technology in Schools Survey and Survey of Principals. Skills demand: Post-secondary Education Participation Survey, Survey of Approaches to Education Planning, Adult Education and Training Survey. Transition between markets: Youth in Transition Survey, National Graduates Survey. Labour market supply: Estimates of educational attainment and activities of population from Census and Labour Force Survey. Direct assessment with the Adult Literacy and Lifeskills Survey.

Slide 6

Elementary/secondary institutional surveys Description Data on enrolments, graduates, finance and teaching staff Collected from provincial/territorial governments, school boards and schools (for private schools) Key strengths Annual Full coverage of K to 12 system Census Source of key indicators: graduation rates, cost per student, teacher-pupil ratio

Slide 7

Elementary/secondary institutional surveys Limitations Timeliness (at the moment) Limited content Some conceptual grey zones to be resolved

Slide 8

Cost per student at the e-s level, 1999-2000 Source: PCEIP

Slide 9

Student-educator ratio, 1999-2000 Source: PCEIP

Slide 10

ICT in School Survey Description Survey of ICT infrastructure (hardware & software) and use within elementary & secondary schools Conducted in 2003 Results in June 2004 Sponsored by Schoolnet program Key strengths Census Current data Could serve as platform for further survey on impact

Slide 11

ICT in School Survey Limitations No information on impact Response rate: 47%

Slide 12

Average number of students per school computer, 2000 Source: PISA

Slide 13

Principals Survey Description University of Montreal/SSHRC sponsored Part of a large-scale project Survey will address impact of change over past 10 years on role of principal Survey will be conducted in the autumn of 2004 Key strengths Linkage to focus groups, other aspects of study Possibility of linking to ICT in School Survey + other institutional data sources

Slide 14

Principals Survey Limitations Sample of about 6000 principals Concern about burden due to number of surveys being done in schools Also: http://crifpe.scedu.umontreal.ca/gtrc/ http://crifpe.scedu.umontreal.ca/gtrc/

Slide 15

Enhanced Student Information System Description Data on enrolment and graduates by program Will cover all public and not-for-profit PS institutions Key strengths Coherence consistent results across PS institutions Program information coded using Classification of Instructional Programs (CIP) Data on student characteristics Linked longitudinally, to follow student pathways through PSE

Slide 16

Enhanced Student Information System Limitations Implementation has been difficult, affecting timeliness Privacy concerns (largely resolved) For-profit postsecondary institutions not covered

Slide 17

Trends in Full-time Enrolment 1985=100 Source: ESIS, CCSIS and USIS

Slide 18

Full-time Enrolment by Sex for 18 to 24 Year Olds Rate of Full-time Enrolment for 18 to 24 Year Olds Source: ESIS and USIS

Slide 19

Postsecondary faculty and finance surveys Description Full-time university faculty survey Part-time university faculty survey College faculty survey University finance survey Tuition and living accommodation survey College finance survey Strengths Long-time series Relevance

Slide 20

Postsecondary faculty and finance surveys Limitations Quality constraints Difficult to compare across PS universe (surveys not integrated)

Slide 21

Slide 22

Constant 2001-02 billions dollars Universities collected record-high revenues in 2001-02, and for the first time in 15 years, revenues from public sources increased at a faster rate than those from private sources. Revenues from private sources (including student fees) Revenues from public sources

Slide 23

Indices of full-time and part-time faculty and full- time equivalent students, 1990=100 Source: Part-time university and college academic staff survey

Slide 24

National Longitudinal Survey of Children and Youth Description Longitudinal sample of children selected from households. Original cohort 25,000 children aged 0-12 started in 1992, followed biennially to age 30. Cross-sectional oversample of 0-5 year olds. Key strengths Data on broad array of motor, cognitive and social factors influencing the level and social distribution of outcomes. Complete families selected to allow for analysis of intra- family effects. Will address questions of impact of early experiences on outcomes later in life.

Slide 25

National Longitudinal Survey of Children and Youth Limitations Sample size limits ability to examine small sub-populations. Household-based sample design precludes direct analysis of school effects.

Slide 26

Vulnerable 19941996 71.1% 56.2% 71.9% 28.9% 28.1% 14.9% 15.7% 13.2% While largely a transitory state, vulnerability touches a significant proportion of children Newly Vulnerable Positive Development Resilient Long term Vulnerable Not Vulnerable

Slide 27

Children with persistent low learning scores have characteristics associated with disadvantage Source : NLSCY, 1994-1995, 1996-1997, 1998-1999

Slide 28

Youth in Transition Survey Description Longitudinal survey of 15 year olds selected from within schools 30,000 students, interviewed biennially starting in 2000 Longitudinal survey of 30,000 18-20 year olds, biennial interviews to age 22-24 Focus on social & educational factors that influence outcomes Key strengths 15 year olds Link to PISA direct skill assessment Contextual information from student, school, home & neighbourhood

Slide 29

Youth in Transition Survey Key strengths -- 18-20 year olds Immediate data on factors influencing high school completion & transitions to PSE & labour market Comparable to 1995 School Leaver Survey Limitations Sample sizes provide insufficient yield of some key sub- populations, such as those who eventually go on to graduate studies

Slide 30

Programme for International Student Assessment Description International school-based skill assessment of 15 year olds In Canada, linked to 2000 YITS same sample PISA focus in 2000 was reading New cohorts in 2003 (focus on math) & 2006 (science) Key strengths Direct proficiency measures Repeated cohorts support trend analysis Proficiency scale for reading skills can be linked to ALL Can analyze impact of a range of school effects International comparisons

Slide 31

Programme for International Student Assessment Limitations As in YITS

Slide 32

Source:Programme for International Student Assessment, 2000. Canada rates near the top of the world in READING literacy

Slide 33

Countries with high achievement in reading also have less variation in achievement by SES (based on PISA, 2000)

Slide 34

In Canada most of the variation in student reading performance is within schools Variation of performance between schools Variation of performance within schools Variation explained by institutional structures Source: PISA 2000

Slide 35

300400500600700800 Reading achievement Proportion of population Public Private Public The apparent private school advantage After controlling for parental SES-private school advantage disappears 400500600700

Slide 36

Educational status of 18 to 20-year- olds no longer in high school

Slide 37

No disability Physical disability Cognitive/ emotional disability The literacy gap between disabled and non-disabled students varies across provinces, particularly for cognitively/emotionally disabled students

Slide 38

Immigrant students overall have lower literacy than other students, but this disadvantage disappears over time

Slide 39

Grade repetition and dropping out are much more frequent for aboriginal students than for other students, even after controlling for socio-economic status

Slide 40

Reading achievement of students in minority language schools

Slide 41

Survey of Approaches to Educational Planning Description First Statcan survey to collect detailed information on how Canadian parents prepare for their childrens postsecondary education First conducted in 1999, collecting information on 20,000 children aged 18 and under Conducted again in October 2002 Key strengths Single source for both financial (ex. current savings, expectations of costs) and non-financial (ex. parents expectations, practices concerning homework) parental factors that may affect childs educational pathway

Slide 42

Survey of Approaches to Educational Planning Limitations Does not trace long-term outcomes.

Slide 43

Source: Survey of Approaches to Educational Planning, 2002 For all income groups except the two highest, parents were most likely to expect around $10,000 in education savings (for parents who were savers when surveyed)

Slide 44

Media coverage [National Post] Grants go to least needy: RESP survey: Parents' expectations too high, study finds [The Vancouver Sun] Families saving more than ever for college education, but it's still not enough: Low-income families are unable to keep pace because federal program disproportionately favours wealthy, study says [GLOBE AND MAIL] More parents dream of university for kids But few expect to be able to save enough to cover full cost of four-year degree

Slide 45

Postsecondary Education Participation Survey Description A new cross-sectional survey of 18-24 year olds (17-24 in Quebec) on postsecondary participation Designed to provide indicators for the Canada Student Loans Program Data collected February 2002 Key strengths Information on cost of attending a postsecondary program Information on sources of revenue to pay for PSE (ex. student loans, family support, jobs)

Slide 46

Postsecondary Education Participation Survey Limitations Small sample. Limited information on non-financial factors influencing postsecondary attendance.

Slide 47

Household income $

Slide 48

Access : No one factor fully accounts for PSE Participation Source: Post-secondary Education Participation Survey, 2002 18-24 year-olds %

Slide 49

Expenditures of CEGEP students vary less than those of college and university students Source: Post-secondary Education Participation Survey, 2002

Slide 50

National Graduate Survey Description Series of cohorts of postsecondary graduates designed to trace participation in advanced studies and labour market success two and five years after graduation Samples are large enough to profile major fields of study by level and province Recent cohorts include Class of 1995 followed up in 2000 and Class of 2000 followed up in 2002 1995 graduates who moved to the US followed up in 1997 and 2000

Slide 51

National Graduate Survey Key strengths High sampling fraction for Masters and PhDs Only source of information on brain drain of postsecondary graduates to U.S. Limitations Does not provide information on non-completers Does not trace long-term outcomes

Slide 52

Indebtedness after graduating from postsecondary education Source: Follow up of Graduates Survey, 2000

Slide 53

Source: Follow up of Graduates Survey 2000

Slide 54

Survey of Earned Doctorates Description New survey to collect information on all doctoral students at the point of convocation Content focus on labour market intentions and mobility Strengths Annual census of doctoral graduates (if funding available) Comparable to U.S. survey Limitations No information on longer-term outcomes

Slide 55

Distribution of doctoral recipients by Field of Study

Slide 56

Adult Literacy and Lifeskills Survey Description Cross-sectional sample aged 16+ with national oversample of in and out of school youth aged 16-25 Direct assessment of skills in prose literacy, document literacy, numeracy and the analytical reasoning component of problem solving; indirect measures of ICT skill and team work Detail on determinants and social, economic and health outcomes Key strengths Direct assessment provides robust measures, which can be linked to factors that determine observed skill profiles to economic outcomes

Slide 57

Adult Literacy and Lifeskill Survey Key strengths Common literacy scales with 1994 International Adult Literacy Survey will allow synthetic cohort analysis, a key to understanding skill attrition Offers international comparisons Limitations Cost of testing limits sample size

Slide 58

Many Canadian adults have weak literacy skills

Slide 59

Canadian high-school leavers less likely to have adequate literacy skills

Slide 60

Source: IALS Prose Literacy by Age

Slide 61

Adult Education and Training Survey Description Conducted every 4 years or so, last done in February 2003 Household-based estimates of demand for training Latest results will be released in April Key strengths Training incidence and intensity, distribution of training, types of training, reasons for taking training, barriers to training, who pays and who gives training Time series

Slide 62

Adult Education and Training Survey Limitations In the past, limited mainly to formal training 2003 survey will have some information on informal and non- formal training Sample size constraints

Slide 63

Canadians living east of Ontario have had lower participation rates in adult learning than the national average with the exception of Nova Scotia in 1997. Differences in provincial participation rates with Canadian average

Slide 64

2006 Census education content New questions will be tested in 2004 Census Test Focus on certification More detail on postsecondary non-university credentials Field of study coded using CIP New question on where highest degree, certificate of diploma was received (country or province) Change to school attendance question (type of institution) Whats being dropped Years of schooling Partial completion of high school or PSE Full-time/part-time split on school attendance

Slide 65

Other surveys with educational content 2001 Postcensal surveys Participation and Activity Limitation Survey Aboriginal Peoples Survey Survey of Labour and Income Dynamics

Slide 66

Use of special education classes varies significantly across provinces and seems to be higher where performance for students with disabilities are lower.

Slide 67

Adjustment to school may be facilitated by pre-elementary programs. Attendance of aboriginal children in pre-elementary programs specifically designed for them has increased steadily since the early 1990s.

Slide 68

When a student lives beyond commuting range of a university, he or she is far more likely to attend college. Source: SLID

Slide 69

New directions in dissemination

Slide 70

More analysis and data On-line report aimed at broad readership Entry point to all educational statistics Research Papers Pan-Canadian Education Indicators CANSIM & Canadian Statistics Access to microdata

Slide 71

Education Matters Bi-monthly 2 feature articles, linked to more in-depth studies Whats new: Access to all Daily releases on education since last issue Link to Research Papers Link to Pan-Canadian Education Indicators

Slide 72

Research Papers Analysis on current issues in education, learning, training and literacy free, on Statcan website Topics to date include access to PSE, national and international student assessments, school-work transitions, adult education, lifelong learning, use of technology, etc Series began November 2002 Already 14 research papers released http://www.statcan.ca/cgi-bin/downpub/listpub.cgi?catno=81-595-MIE

Slide 73

Pan-Canadian Education Indicators Program (PCEIP) Joint venture of Statistics Canada and the Council of Ministers of Education, Canada (CMEC) Aimed at policy makers, practitioners and the public Tables will be updated every 6 months, accessible through Education Matters Report available on-line at no charge Tables can be downloaded http://dissemination.statcan.ca/english/freepub/81-582-XIE/2003001/educ.htm

Slide 74

International Adult Literacy Survey database Interactive data tool permitting users to retrieve prose, document and quantitative literacy data for each participating country wide range of combined intermediate variables Offers estimates of mean performance and percentages of all three IALS literacy scales Easy to use Free on Statcan website: http://www.statcan.ca/english/freepub/89-588-XIE/free.htm

Slide 75

CESC-SSHRC Education Research Initiative Promotes policy-relevant research in education, draws on Statcan and CMEC data 9 research projects funded in 2003 on the theme of learning outcomes Another 9 just selected on the themes of transitions and, again, outcomes www.sshrc.ca/web/apply/program_descriptions/ine/education_research_e.afpwww.sshrc.ca/web/apply/program_descriptions/

Slide 76

CANSIM and CTCES Common Data Repository Historically, limited education presence in CANSIM & Canadian Statistics No central location for holding education data: Difficult to integrate data from different sources No consistency in metadata Redefine output & distribute via appropriate vehicle CANSIM Canadian Statistics Self-contained E-Products

Slide 77

CANSIM and CTCES Common Data Repository Currently developing and populating Common Data Repository (CDR) Stocking data in common framework For education: 28 surveys to incorporate Defining new intelligent output New online analytical tools (internal) Bridge to CANSIM & Canadian Statistics Ultimately, dynamic publishing

Slide 78

Microdata access Public use microdata files Research Data Centres Remote access

Slide 79

Public use microdata files NGS/FOG Looking at feasibility YITSN PISAY PEPS Looking at feasibility SAEPY AETSY IALSY ALLY

Slide 80

Research Data Centres Main access point for longitudinal datasets, no suppression of analytically useful variables Proposals vetted by SSHRC Access restricted to facilities in universities Researchers provide STC with an analytical product No charge to researcher Important associated activities: workshops to familiarize researchers with datasets, training in analytical techniques

Slide 81

Remote access CES offers remote access Designed to complement Research Data Centres Researchers submit code to STC via Internet (having tested it against a dummy file) Output vetted for confidentiality is provided electronically within 2 working days