Innovative assessment approaches and new paradigms … · assessment of teachers‘ professional competencies ... TEDS-M coming from the cognitive perspective, ... without using the

Gabriele Kaiser – University of Hamburg

Innovative assessment approaches and new paradigms within the

assessment of teachers‘ professional competencies

Gabriele Kaiser (University of Hamburg, Australian

Catholic University)

1


Since late 1990s a growing number of empirical studies on professional competencies of future or practising teachers; often case studies, carried out in own institutions. Focus on mathematics teachers.

Nowadays a huge increase of research, many small-scale and a few large-scale studies.

2


Current trends within discussion on teacher education and practising teachers

Identification of three trends of mathematics teacher education by Krainer and Llinares (2010):

- First trend: increasing attention to the social dimension of teacher education, i.e. concept of collaborative learning, teacher-inquiry groups, communities of practice ….

- Second trend: focus on teachers‘ reflective practice, i.e. teachers‘ noticing within teaching, change of practice …

- Third trend: increasing attention to the general conditions of teacher education, e.g. structure, institutional setting, human resources …

3


Introduction Krainer and Llinares (2010) argue:

• It is necessary “to take these three strands seriously and regarding them as challenges for the future.”

Currently, central dichotomy of research on teacher and their education, namely studies are either oriented towards cognitive aspects or towards situated aspects.

The first two trends reflect a situated perspective, the last trend a cognitive perspective.

4


Several large-scale studies on mathematics teacher knowledge share as common theoretical orientation a cognitive perspective referring to the approach of professional knowledge of teachers by Shulman (1986) distinguishing between different knowledge facets functional for teaching:

• IEA-Study Teacher Education and Development Study in Mathematics (TEDS-M) (Blömeke et al. 2014) and its predecessor Mathematics Teaching in the 21st Century (MT21) (Schmidt et al., 2011)

• Mathematics Teaching and Learning to Teach Project (MTLT) by the University of Michigan (Loewenberg Ball & Bass, 2000)

• Cognitive Activation in the Classroom Project (COACTIV) by German researchers (Kunter et al., 2013).

5


However, new studies have been developed in in the last decades, characterised by a situated perspective, departing from the concept of teachers‘ noticing and research on teachers‘ expertise.

Examples:

various Follow-up studies of TEDS-M such as TEDS-FU and TEDS-Instruct (Kaiser et al., 2015)

Learning to Learn from Mathematics Teaching Project (Santagata et al., 2011)

Classroom Video Analysis Instrument (Kersting et al., 2012)

6


Aim of the lecture Analysis of the relationship between cognitive and situated approaches in

empirical studies of teachers‘ professional competencies.

Two questions:

(1) What are characteristics of frameworks and instruments that are adequate for the usage of one or both perspectives and what are the benefits and limitations exhibited by these frameworks and instruments?

(2) Can these complementary perspectives be integrated within one theoretical framework with joint assessment instruments of teachers’ professional competencies, including finally their teaching?

7


1. Review of the state-of-the art on teachers‘ professional competencies –

theoretical frameworks and evaluation instruments


Current situation concerning the theoretical framework on teachers‘ professional knowledge Common core of most studies refer to Shulman‘s description of pedagogical

content knowledge (PCK) (1987) as

„special amalgam of content and pedagogy that is uniquely the province of teachers, their own special form of professional understanding”.

However, no general consensus on the facets of this concept (survey by Depaepe, Verschaffel, Kelchtermans, 2013).

Two different paradigms concerning the conceptualisation of PCK:

“whether mathematical knowledge in teaching is located ‘in the head’ of the individual teacher or is somehow a social asset, meaningful only in the context of its applications” (Rowland & Ruthven, 2011).

9


Consequences of this cognitive and situated perspective on the detailed definition of PCK according to Depaepe et al. (2013):

Cognitive perspective: identifies a limited number of components to be part of PCK and distinguishes PCK from other categories of teachers’ knowledge base;

Situated perspective: describes PCK as multi-dimensional, teachers’ choices simultaneously reflect mathematical and pedagogical deliberations.

10


Consequences for the investigation of PCK:

Cognitive perspective: PCK measured independently from the classroom context in which it is used, most often through a test, relations to other facets of teachers’ knowledge evaluated;

Situated perspective: assume that investigating PCK only makes sense within the context in which it is enacted. Therefore, the studies often rely on classroom observations supplemented by other data sources.

Depaepe et al. (2013) point out that both perspectives are needed, because both have pitfalls and shortcomings as well as strengths.

In the following research is described, which intends to integrate both kinds of research, TEDS-M coming from the cognitive perspective, TEDS-FU bringing in a situated approach, TEDS-Instruct as integrating project.

11


2. The TEDS-M study as example of the cognitive approach – its framework

and its instruments


Carried out under the auspices of the International Association for the Evaluation of Educational Achievement (IEA); Evaluated internationally the effectiveness of teacher education in terms of teacher knowledge and teacher beliefs across countries and subject-specifically (overview in Blömeke, Hsieh, Kaiser, Schmidt, 2014; Tatto et al., 2008).

13

Teacher Education and Development Study: Learning to Teach Mathematics (TEDS-M)


First large-scale assessment of higher education with direct testing covering graduates from 16 countries from East and West;

Focus of TEDS-M: future teachers in their final year of teacher education:

Primary study: future teachers aiming for grade 1 through 4

Secondary study: future teachers aiming for lower-secondary grades

Two studies based on nationally representative samples, follow the rigorous IEA quality control mechanisms of sampling, data collection, coding, and data analysis.

About 23,000 prospective teachers participated in the two studies, which took place from 2008-2009, results released in 2010.

14


Theoretical framework

Theoretical starting point of TEDS-M are teaching abilities – called ‘professional competencies’ divided into cognitive facets (teachers’ professional knowledge) and affective-motivational facets (professional beliefs).

Referring to Shulman (1986) the professional knowledge of teachers are divided into several facets: mathematics content knowledge (MCK), mathematics pedagogical content knowledge (MPCK), including curricular knowledge, and general pedagogical knowledge (GPK).

Concerning professional beliefs in particular epistemological beliefs about the nature of mathematics and beliefs about the teaching and learning of mathematics were included.

15


Conceptual model of teachers’ professional competencies

16

Affective - motivational characteristics: Professional beliefs,

motivation and self-regulation

Teacher competencies

Cognitive abilities: Professional knowledge

Content knowledge

Beliefs about mathematics and the teaching and

learning of mathematics

Professional motivation and self-regulation

Pedagogical content

knowledge General pedagogical

knowledge ( Shulman

1986 ) (Richardson 1996; Thompson 1992)


Assessment Instruments:

Multiple instruments such as instruments of a prospective teacher survey, teacher educator survey, expert survey, document analysis of a sample of course offerings.

Competencies of future teachers’ MCK and PCK, GPK only in Taiwan, US and Germany

Assessment with a paper-and-pencil test, lasting 90 minutes. Variety of closed and open items or items with short-answers.

17


Released sample item: Outcome: MCK / Domain: Algebra / Sub-domain: Reasoning / Secondary level

Prove the following statement: If the graphs of linear functions

and

intersect at a point P on the x-axis, the graph of their sum function

must also go through P.

18

= +( )f x ax b = +( )g x cx d

+( )( )f g x

Possible correct answer: Response carefully lays out the steps of the proof in a general way, without using the given formulas of f(x) and g(x). Example: Suppose f(x) and g(x) intersect at point (p, 0) on the x-axis. Then f(p) = 0, g(p) = 0. Then (f + g)(p) = f(p) + g(p) = 0 + 0 = 0.


Outcome: MPCK / Domain: Geometry / Sub-domain: Curriculum/Planning / Primary level When teaching children about length measurement for the first time, Mrs. [Ho] prefers to begin by having the children measure the width of their book using paper clips, then again using pencils. Give TWO reasons she could have for preferring to do this rather than simply teaching the children how to use a ruler?

Significant and acceptable reasons Reason 1: Understanding of what measurement is Reason 2: Need for standard units Reason 3: Choosing most appropriate unit


Detailed results are available (e.g. Blömeke et al., 2014), concentration on selected central results from secondary study.

20


Unexpected results: • Taiwanese future teachers outperform all

other countries • Given the HDI, the performance

of Russian and Polish future teachers is remarkable.

• US American future teachers around the international mean, whereas German future teachers above.

• Difference to Taiwan for both countries about 1.5 SD, but there is also a big difference to Russia and Singapore. In addition, significantly lower performance than Poland and Switzerland.

21

* Reduced Coverage, Combined Participation Rate < 75% or other limitations

Gabriele Kaiser – University of Hamburg 22

USA around the international mean, Germany significantly above. Outstanding results of Taiwanese future teachers, followed by future teachers from Russia, Singapore and Switzerland, followed by Germany.


Country-specific profiles concerning the importance of mathematics content knowledge in comparison to pedagogical content knowledge Three country-specific achievement profiles:

• Higher achievements in mathematics than in pedagogical content knowledge: from Asia Singapore, Taiwan and Malaysia, from East and Middle Europe Russia, Poland, from Africa Botswana

• Higher achievements in pedagogical content knowledge than in mathematics: Germany, Norway, Switzerland, as well as Chile and Georgia

• Knowledge relatively levelled: USA, Thailand, Oman, Philippines

23


Profiles independent of absolute level of achievement:

countries with higher-achieving future teachers can be found in both distinct groups, e.g. Taiwan or Switzerland, same for lower-achieving countries.

Cultural explanations: Strong influence of cultural traditions, which shape profiles: • the cultural tradition of the Confucian Heritage Culture dominant in East

Asian countries with high importance of content knowledge in teacher education,

• in Western countries various traditions, e.g. content-related approaches with high emphasis on content knowledge but within pedagogical content knowledge in Continental European didactic traditions (e.g. in Germany, Switzerland and Eastern Europe), child-centred approaches within progressive education (e.g. Scandinavian and North American countries).

24


3. TEDS-FU as example of the situated approach


Implementation of a Follow-up study of TEDS-M, TEDS-FU, in 2010

Main question:

How can professional competence be analysed in a more performance-oriented way including teacher expertise building on the theoretical framework and the instruments of TED-M?


Enrichment of theoretical framework of TEDS-FU: Integration of additional practice-oriented, situated facets linked to the

concept of noticing, which takes up a strong action-oriented or instrumental point of view apart from the knowledge-based facets of teacher competence, namely MCK, MPCK and GPK, from TEDS-M.

Three situated facets of noticing (Carter et al., 1988; Van Es & Sherin, 2002; Sherin, Jacobs & Philipp 2011):

- Perceiving particular events in an instructional setting or “attending to particular events”

- Interpreting the perceived activities in the classroom or “making sense of events in an instructional setting”

- Decision-making, either as anticipating a response to students’ activities or as proposing alternative instructional strategies (Erickson, 2011; Neuweg, 2011)

27


Connection to new conception on teachers‘ professional competencies as continuum (Blömeke, Gustafsson, Shavelson, 2015)

28


TEDS-FU was carried out from 2010 to 2013, web-based testing.

Participants from the TEDS-M primary and secondary study were tested on a voluntary basis: 171 teachers from the secondary cohort and 130 teachers from the primary cohort.

Assessment of the professional competencies of the teachers using video vignettes with short teaching sequences dealing amongst others with effective classroom management, heterogeneity, individualisation, teaching strategies, continuation of the teaching sequences with possible teaching options; usage of scripted video clips.

29


Design of TEDS-FU

30


Example of a shortened video vignette

Context information: In the video a sequence taken from a mathematics classroom in a German comprehensive school is shown. Concerning their mathematical abilities the class is heterogeneous.

Mathematical background: The task dealt with is the following: On a farm there are geese and cows. Altogether there are 105 animals, and

there are four times as much geese than cows on the farm. How many geese and how many cows are on the farm?

31


Video

32


Variety of items on MPCK and GPK using Likert scales and open items, e.g.:

Likert scales MPCK (perceiving & interpreting): Assessment of student's answers from a mathematical point of view

Open items GPK (decision-making): Prevention of classroom disturbances Open items MPCK (perceiving & interpreting): Rephrasing of a student's

argument / Description of the student's misconception of variables Background concerning MPCK: well-known misconception of variables, first

published by Rosnick & Clement (1980), variable understood as entity and not as quantity of the entity or no distinction between entity and variable.

New item format allow insight into possible actions of teachers, i.e. into their possible performance.

33


Data evaluation Development of extensive coding manuals with correct, semi-correct and

wrong answers out of the material using verbatim answers by test persons.

Example item: Kira's answer 4g = k is not correct if k refers to cows and g refers to geese.

Kira's answer is possibly based on a misconception of variables. Please describe this misconception.

Example of a correct answer:

"She translated the wording 'four times more geese' literally into the mathematical language rather than multiplying the number of cows. So she uses the variable as a mere substitute in the spoken language."

34


Selected results

Interesting insight into the structure and development of professional knowledge of early career teachers:

MCK: Significant decrease of average level of MCK between the first testing in 2008 at the end of their teacher education within TEDS-M and 2012 within TEDS-FU; expected result.

MPCK: average level of MPCK remained stable; surprising result as a decrease would have been plausible due to the nature of the paper-and-pencil test assessing partly declarative knowledge.

Points to the relevance of practical experience as learning opportunity for the development of MPCK, which is stated by the research on expertise for other professions already for a long time.


Analysis of the rank order of the participants regarding MCK and MPCK in 2008 and 2012 yields interesting differences between MPCK and MCK:

in MCK the rank order remains nearly unchanged, i.e. the knowledge level of the prospective teachers at the end of their education predicts very strongly the achievement level after four years of teaching practice.

In MPCK varied results: the level of MPCK at the end of teacher education predicts significantly the level of MPCK after four years of teaching, but the rank order of the mathematics teachers is less stable in this knowledge facet than in MCK.

Tentative conclusion referring to expertise research: MPCK of young teachers at the beginning of their career more flexible; teaching experience as a strong opportunity to learn, influencing MPCK stronger than MCK.

36


Selected results

Analysis of nature of MPCK and GPK evaluated via digitalised paper-pencil-test and situated video-based tests: detailed analysis at item level (using median split) point to the necessity of a combination of both evaluation methods.

Situated, practice-oriented evaluation of GPK and MPCK identifies stronger the commonalities of both domains in contrast to digitalised paper-pencil-test focusing stronger on differences in theory-based knowledge.

37


4. Professional Competencies of Mathematics Teachers – Integration of

Cognitive and Situated Approaches


Unexpected results on the nature of teacher expertise within TEDS-FU– describing the relation between 1) knowledge and 2) perceiving, interpreting and acting in classroom situations, and 3) the speed of student error recognition.

Distinguishing the facets of perceiving, interpreting and acting in classroom situations under an applied perspective, i.e. either content-related or pedagogical-oriented, analyses point out that teacher expertise can neither be adequately described via models claiming either homogeneity of these indicators for expertise nor by distinctions of facets according to domains or assessment methods.

Based on our data, expertise can best be described with a two-dimensional model distinguishing between content-related knowledge (MCK, MPCK and speed in mathematics error recognition) and performance-related competencies (GPK, perceiving, interpreting and acting).

39


5. Prospects: Professional Competencies of Mathematics Teachers and their

Relation to Students‘ Achievements and the Inclusion of Societal Aspects


Further developments: Necessity of inclusion generic societal aspects (Blömeke & Kaiser, 2016)

41


Third part of the triad of TEDS studies: connection of the professional competencies of teachers with growth of students‘ achievements Identification of patterns between relation of teachers‘ competence and

promotion of students‘ learning

Study: March 2014-March 2019

Two federal States of Germany: Hamburg als metropolitan area with high number of migrant students, Thuringia and Saxonia as more rural area from former GDR

Implementation of the framework and the instruments of TEDS-FU with teachers teaching students from grade 7 to grade 9 and extensive classroom observations focusing on teaching quality.


Impact model of TEDS-Instruct and TEDS-Validate

43


Extension to Asia – Study East-West

Study by Xinrong Yang from Southwest University (China) & University of Hamburg (Germany):

How far can the studies TEDS-Instruct / TEDS-Validate developed and implemented in a Western country be implemented into an East Asian country?

In detail:

How far are the structural relations between mathematics teachers’ professional competence and their students’ mathematical achievements and its longitudinal development taught by them over one year in China and Germany similar?

44

TEDS East-West


Preliminary results

Detailed results available in autumn 2017, 220 teachers participated in the competence measurement in Chongqing, metropolitan area in China.

Most parts of the instruments on teachers‘ professional competences can be transferred to East Asia, but competence measurement had to be done differently, as paper-pencil-test with integrated videos, done within an in-service-activity in contrast to Germany (individualised testing, web-based at home).

Instructional quality and its measurement difficult to transfer, apparently understood and therefore conceptualised differently in East and West.

45


First results of knowledge tests GPK test not adequate for TEDS-East-West, too strong focus on Western topics

such as classroom management, therefore reduction to a few items concentrating on lesson planning procedures, interaction in teaching and assessment.

MCK and MPCK test: high number of missing data in TEDS-Instruct, probably due to time limitation and/or lack of motivation, in contrast to TEDS-East-West. Only low number of joint items allowing comparisons.

MCK: higher achievements of Chinese teachers (influenced by high number of missing data from German teachers), average solution frequency by Chinese teachers 86.69% in contrast to 62.32% by German teachers.

MPCK: still higher achievements of Chinese teachers, average 80.37% (Chinese teachers) versus 58.70% (German teachers).

46


First results of knowledge tests

Items with exceptional differences refer in MCK to mathematical content, highly familiar to the Chinese teachers like algebra or geometry at school level, in MPCK to students‘ proofs.

Items with smaller differences refer concerning MCK to for Chinese teachers less familiar mathematical topics such as statistics or number theories, MPCK to students‘ thinking processes

Results of the video-based testing will be available soon.

47


Summary

Development of the discussion on the professional competencies of teachers point out the necessity to:

reflect about the possibility to transfer Western frameworks and instruments to other cultures, not as easy as assumed

integrate knowledge-based measurement instruments with more practice-oriented instruments

include instructional quality as central mediator variable

combine teachers‘ professional competencies with students‘ growth as central aim of educational activities in order to answer the „so What-question by many scholars and politicians.

48


Thank you very much for your attention.

49


Back up

50

Documents

Innovative assessment approaches and new paradigms … · assessment of teachers‘ professional competencies ... TEDS-M coming from the cognitive perspective, ... without using the