What Works Best?

Expert and Novice Teachers‘ Beliefs

About School Effectiveness

Johanna Fleckenstein1, Friederike Zimmermann2, Jens Möller2 & Olaf Köller1

1Leibniz Institute for Science and Mathematics Education (IPN), Kiel 2Institute for Psychology in Education, University of Kiel

April 7, 2014

2014 Annual Meeting of the American Educational Research Association (AERA)

April 3 - 7, 2014 in Philadelphia, Pennsylvania

Leibniz Institute for Science

and Mathematics Education

…regulate teachers‘ behavior and decisions in the classroom (Köller, Baumert & Neubrand, 2000; Pajares, 1992; Schoenfeld, 2000, 1983)

…influence student achievement (Dubberke et al., 2008; Staub & Stern, 2002)

…develop at an early stage in life (Lortie, 1975; Buchmann, 1987; Pajares,

1992; Wilson, 1990)

…are very persistent (Abelson, 1979; Lewis, 1990; Nespor, 1987; Nisbett & Ross,

1980; Posner et al., 1982; Rokeach, 1968; Helmke, 2003)

…can differ significantly from what is conveyed in teacher

education and professional development (Wahl, 2002)

…may diverge from empirical findings (Kunter & Pohlmann, 2009)

Teachers‘ Beliefs…

2

What Works in School?

3

Small

Classes!

believe in

know about

Direct

Instruction!

Open

Learning!

Students‘

Motivation!

(1) To what extent do teachers’ beliefs diverge from

findings of empirical research (cf. Hattie, 2009)?

(2) Are there differences in the beliefs of novice and

expert teachers?

Research Questions

4

• Milestone in school-effectiveness research

• Synthesis of meta-analyses concerning 138 factors of

successful learning

• Cohen‘s d as a measure for practical significance

• d > 0.4 as the point of reference for substantial effects

– greater than one year of average schooling (Köller, 2012)

Hattie‘s Meta-Meta-Analysis (2009, 2012)

5

• Sample:

– N = 371 pre-service teachers (M.Ed.)

• Age: M = 25.03 (SD = 2.94)

• 72% female

– N = 358 in-service teachers

• Age: M = 51.25 (SD = 10.09)

• 59% female

• Questionnaire: The Hattie-o-meter

• Estimation of the effect size d for 16 factors on a

scale from d = -0.4 to d = 1.0

Methods

6

(1) To what extent do teachers’ beliefs diverge from

findings of empirical research (cf. Hattie, 2009)?

(2) Are there differences in the beliefs of novice and

expert teachers?

Research Questions

7

Results: Descriptives

8

Factors dHattie MExperts MNovices

Feedback .73 .55 (.23) .62 (.24)a

Meta-cognitive strategies .69 .53 (.25) .53 (.26)

Prior achievement .67 .32 (.23) .39 (.24)a

Professional development .62 .40 (.23) .42 (.25)

Direct instruction .59 .28 (.23)a .18 (.25)

Motivation .48 .63 (.24) .73 (.22)a

Expectations .43 .36 (.25)a .23 (.29)

Self-concept .43 .55 (.21)a .48 (.24)

Attitude .36 .56 (.24) .66 (.21)a

Frequent/effects of testing .34 .34 (.24) .34 (.26)

Class size .21 .34 (.31) .59 (.30)a

Co-/team teaching .19 .37 (.27) .45 (.28)a

Within-class grouping .16 .48 (.27) .61 (.24)a

Problem-based learning .15 .52 (.23) .52 (.25)

Multi-grade/age classes .04 .19 (.26) .22 (.27)

Open learning .01 .29 (.27) .37 (.27)a

Results: Descriptives

9

Factors dHattie MExperts MNovices

Feedback .73 .55 (.23) .62 (.24)a

Meta-cognitive strategies .69 .53 (.25) .53 (.26)

Prior achievement .67 .32 (.23) .39 (.24)a

Professional development .62 .40 (.23) .42 (.25)

Direct instruction .59 .28 (.23)a .18 (.25)

Motivation .48 .63 (.24) .73 (.22)a

Expectations .43 .36 (.25)a .23 (.29)

Self-concept .43 .55 (.21)a .48 (.24)

Attitude .36 .56 (.24) .66 (.21)a

Frequent/effects of testing .34 .34 (.24) .34 (.26)

Class size .21 .34 (.31) .59 (.30)a

Co-/team teaching .19 .37 (.27) .45 (.28)a

Within-class grouping .16 .48 (.27) .61 (.24)a

Problem-based learning .15 .52 (.23) .52 (.25)

Multi-grade/age classes .04 .19 (.26) .22 (.27)

Open learning .01 .29 (.27) .37 (.27)a

d ≥ .40 d < .40

• Aggregated profile correlation per group

Results: Profile Correlations

10

Expert teachers Novice teachers

Mr SDr Mr SDr

.21 .31 .05 .28

t[726] = 7.07; p < .001; d = .54

(1) To what extent do teachers’ beliefs diverge from

findings of empirical research (cf. Hattie, 2009)?

(2) Are there differences in the beliefs of novice and

expert teachers?

Research Questions

11

• Substantial improvement in goodness-of-fit indices: (Cheung & Rensvold, 2002)

ΔCFI > .01; ΔRMSEA > .015

Results: CFA

12

A priori: 3-dimensional Empirical: 4-dimensional

Student Student

School School

Teaching Teaching

Achievement

χ2[95]=586.36; CFI=.85;

RMSEA=.08; TLI=.81; SRMR=.08

χ2[92]=340.85; CFI=.92;

RMSEA=.06; TLI=.90; SRMR=.05

Teaching Achievement Structure Student

Feedback Direct instruction Multi-grade/age

classes

Motivation

Meta-cognitive

strategies

Expectations Open learning Prior achievement

Professional

development

Frequent/effects of

testing

Class size Self-concept

Problem-based

learning

Co-/team teaching Attitude

Within-class

grouping

Results: CFA

13

Achievement Structure Student

Teaching .41** .70** .66**

Achievement -.16** .17*

Structure .78**

**p < .01; *p < .05

• Correlation matrix:

• Partial scalar invariance (strong invariance)

• Mean group differences in latent variables:

Results: Structured means analysis

14

Factor MΔ p

Teaching -0.12 ns

Achievement 0.67 <.001

Structure -0.56 <.001

Student -0.70 <.001

z-standardized; MNovices = 0; MExperts = MΔ

• Discrepancy between teachers‘ beliefs and research

findings

• Stronger overall congruence with research findings for

the expert teachers

• Novice teachers

– Infra- and surface-structural conditions of schooling

– Student-internal variables

– Student-centered, progressive education

• Expert teachers

– Teacher as a central figure in the classroom

– Achievement-related variables

Discussion

15

• Educational researchers…

…should make their research findings more available to

teachers

• Teacher educators…

…should familiarize teachers with findings of school

effectiveness research

…should challenge them to continuously reflect on their

own beliefs

• Teachers…

…should stay in touch with research communities

Implications

16

Thank you for your

attention!

Johanna Fleckenstein

Leibniz Institute for Science and Mathematics Education (IPN)

Olshausenstr. 62, D-24114 Kiel

Tel: +49 431 880-1309

E-Mail: jmoeller@ipn.uni-kiel.de

Leibniz Institute for Science

and Mathematics Education

Appendix

18

Leibniz Institute for Science

and Mathematics Education

factors teaching achievement structure student

feedback 0.621 - - -

meta-cognitive strategies 0.681 - - -

professional development 0.704 - - -

problem-based learning 0.692 - - -

direct instruction - 0.521 - -

expectations - 0.688 - -

frequent/effects of testing - 0.484 - -

multi-grade/age classes - - 0.368 -

open learning - - 0.653 -

class size - - 0.512 -

co-/team teaching - - 0.555 -

within-class grouping - - 0.767 -

motivation - - - 0.666

prior achievement - - - 0.367

self-concept - - - 0.505

attitude - - - 0.573

Appendix: Factor loadings matrix

19

• Expert(-novice) paradigm

• Professional competence = Individual features that enable

professional behavior of teachers (Baumert & Kunter, 2006)

• Professional competence is acquired and developed further

by continuous training and experience (Bromme, 2008)

• Comparisons of experts and novices (Berliner, 2001, 2004)

Professional Competence of Teachers

Beliefs Knowledge

Motivation

Self-

Regulation

20

• School effectiveness research

– What works in school?

– Determinants of student achievement

• Hattie‘s meta-meta-analysis as a milestone

• Findings are congruent with prior research

• Impact on teaching reality

– Evidence-based practice

– Teachers have to know about and believe in empirical evidence

What do teachers believe works in school?

Do their beliefs differ from research findings (Hattie, 2009)?

Objective: What Works Best?

21

• “Please estimate the effect of each of the factors

below on students’ achievement”

• Sample Items:

Questionnaire

22

• Descriptive analysis of group means

• Profile correlations (Pearson‘s r) of the teachers‘

ratings with the distribution of Hattie‘s d‘s

• Group comparisons on a latent level using

confirmatory factor analysis (CFA)

– Multiple group modeling approach (Meredith & Teresi, 2006)

– Partial scalar invariance as the minimum

requirement for the comparison of means (Byrne, Shavelson & Muthén, 1989)

Statistical Analyses

23