EARLI SIG 1: CONFERENCE PROGRAM

Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012

1EARLI SIG 1ASSESSMENT AND EVALUATION

CO N F E R E N C E P R O G R A M

E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N

Linking Multiple Perspectives on

Assessment

28-31 August, 2012 Brussels


2

SIG 1 COORDINATORSDominique Sluijsmans

Katrien StruyvenErnesto Panadero

CONFERENCE PROGRAMSixth biennial meeting of Earli Sig 1: Assessment and Evaluation

"Linking Multiple perspectives on Assessment"


3

Mai

n co

nfer

ence

room

Oly

mpi

a Bo

urse

Ba

llroo

m

Brea

kout

room

1

Euro

Brea

kout

room

1

Ster

ling

Brea

kout

room

1

Dol

lar

Brea

kout

room

1

Yen

10h30

Ope

ning

11h-12h30

Keyn

ote

(p.6)

Jose

ph K

esse

ls12h30-

14hRe

stau

rant

Mid

twon

Gril

l

14h- 15h30

Sym

posi

um (p

.6)D

avid

Bou

dRe

fram

ing

feed

back

: ne

w p

ersp

ectiv

es, n

ew

dim

ensi

ons.

Jenn

y Le

nkei

t (p.9

)H

ow e

ffect

ive

are

educ

atio

nal

syst

ems?

A v

alue

-add

ed a

ppro

ach

to st

udy

tren

ds in

PIR

LS.

Khal

ed B

arka

oui (

p.10)

Effec

ts o

f Key

boar

ding

Ski

lls

on P

erfo

rman

ce o

n Co

mpu

ter-

Base

d L2

Writ

ing

Test

s.

G. B

row

n an

d A

. Rem

esal

(p.12

)Sp

anis

h as

For

eign

Lan

guag

e Te

ache

rs’ c

once

ptio

ns o

f as

sess

men

t: pr

elim

inar

y re

sults

from

an

inte

rnet

inqu

iry

Jose

ph C

onbo

y (p

.14)

Teac

her C

omm

unic

atio

n of

Ev

alua

tion

Resu

lts: I

mpa

ct

on S

tude

nt E

ngag

emen

t in

Sch

ool.

Dan

iel C

aro

(p.9)

A N

ew Lo

ok a

t the

Eva

luat

ion

of S

ocio

logi

cal T

heor

ies i

n In

tern

atio

nal L

arge

Sca

le

Educ

atio

nal A

sses

smen

ts.

Ast

rid

Birg

itte

Egg

en (p

.11)

Dev

elop

ing

natio

nal s

tand

ards

fo

r the

ass

essm

ent o

f writ

ing

in

com

puls

ory

educ

atio

n.

Gav

in B

row

n (p

.13)

Asse

ssm

ent i

n Br

azili

an h

ighe

r ed

ucat

ion:

Val

idat

ion

of a

Po

rtug

uese

ver

sion

of t

he S

tude

nts’

Conc

eptio

ns o

f Ass

essm

ent (

SCoA

-VI

) inv

ento

ry.

Robi

n Ti

erne

y (p

.15)

Thin

king

abo

ut F

airn

ess i

n Cl

assr

oom

Ass

essm

ent.

Alis

on G

ilmor

e (p

.10)

Revi

sion

ing

Nat

iona

l Ass

essm

ent

in N

ew Z

eala

nd.

Khal

ed B

arka

oui (

p.12)

Exam

inin

g Ta

sk E

quiv

alen

ce

in L

2 W

ritin

g Te

sts:

A M

ixed

M

etho

ds A

ppro

ach.

Wei

Shi

n Le

ong

(p.14

)Co

ncep

tions

of C

lass

room

As

sess

men

t: Fi

ndin

gs fr

om U

sing

Q

-Met

hodo

logy

.

Caro

l Eva

ns (p

.16)

The

emot

iona

l dim

ensi

on

of fe

edba

ck.

15h30-16h

Coffe

cor

ner

16h-17h30

Roun

dtab

le (p

.16)

Mai

n co

nfer

ence

room

Oly

mpi

a Bo

urse

Bal

lroom

Wor

ksho

p 1

(p.25

)G

odfr

ey P

ell,

and

Rich

ard

Fulle

rTh

e ch

alle

nge

of m

eetin

g th

e as

sess

men

t nee

ds o

f bot

h co

mpe

tent

and

und

erpe

rfor

min

g st

uden

ts w

ithin

exi

stin

g ed

ucat

iona

l str

uctu

res:

Intr

oduc

ing

sequ

entia

l tes

ting?

Le

sson

s for

pol

icy

and

prac

tice.

Wor

ksho

p 2

(p.26

)Ed

uard

o G

arcí

a Ji

mén

ez,

Mª S

oled

ad Ib

arra

Sái

z an

d G

rego

rio R

odríg

uez

Góm

ezEv

alCO

MIX

: A w

eb se

rvic

e fo

r u-

feed

back

and

u-fe

edfo

rwar

d.

Wor

ksho

p 3

(p.26

)N

iek

van

Bent

hum

A th

eory

of i

mpr

ovem

ent f

or

teac

hers

’ pro

fess

iona

l dev

elop

men

t in

Ass

essm

ent f

or Le

arni

ng

Gav

in B

row

n. (p

.16) A

ccur

acy

in S

elf-

asse

ssm

ent:

Deb

atin

g th

e M

erit

of A

ccur

acy

whe

n St

uden

ts E

valu

ate

thei

r Ow

n W

ork.

Pa

tric

k La

i. (p

.17) I

ntro

duct

ion

of q

ualit

y as

sura

nce

elem

ents

into

the

asse

ssm

ent p

olic

y of

gen

eric

gra

duat

e at

trib

utes

of u

nive

rsity

usi

ng th

e co

urse

-em

bedd

ed a

sses

smen

t m

etho

ds.

Hui

yong

Sea

h-Ta

y. (p

.18) W

hat T

ypes

of A

sses

smen

t Con

text

s Sup

port

Sel

f-Reg

ulat

ed Le

arni

ng?

Kare

l Van

Nie

uwen

huys

e. (p

.18) T

he re

latio

nshi

p be

twee

n pa

st a

nd p

rese

nt in

end

of t

erm

test

s in

Flem

ish

hist

ory

educ

atio

n.M

adda

lena

Tar

as. (

p.19)

Und

erst

andi

ng o

f Ass

essm

ent b

y U

nive

rsity

Lect

urer

s: ho

w th

is li

nks t

o th

eorie

s.A

mir

a Ro

m a

nd M

icha

l Nac

hsho

n. (p

.20) H

igh

Scho

ol S

tude

nts P

ortfo

lios i

n Sc

ienc

e fo

r All:

Indi

cato

rs o

f the

Qua

lity

of th

eir T

each

ers W

ork.

Núr

ia d

e Sa

lvad

or. (

p.20)

Form

ativ

e As

sess

men

t usi

ng C

olle

ctiv

e Po

rtfo

lios.

Mas

ahir

o A

rim

oto.

(p.21

) Lin

king

a fr

amew

ork

of fo

rmat

ive

asse

ssm

ent a

nd e

valu

atio

n fo

r “le

arni

ng to

lear

n” to

lear

ning

circ

les f

or tr

ansf

orm

atio

nal s

choo

l lea

ders

hip.

Rita

Hea

ding

ton.

(p.21

) Exp

lorin

g un

derg

radu

ate

stud

ents

’ inf

orm

al, f

orm

ativ

e pe

er a

sses

smen

t net

wor

ks .

Ho

Yi X

ian.

(p.22

) Stu

dent

s’ un

derp

erfo

rman

ce d

ue to

pse

udo

unde

rsta

ndin

g of

mat

hem

atic

al co

ncep

ts: W

hat c

an b

e im

prov

ed in

scho

ol a

sses

smen

t?St

even

Nijh

uis.

(p.23

) Will

dig

itally

aid

ing

the

crea

tion

of a

pro

ject

pla

n cr

eate

tran

sfer

?Kr

isti

en C

arne

l. (p

.23) U

ITZI

GT

inte

grat

ed a

sses

smen

t of c

ompe

tenc

es o

n ou

tflow

leve

l in

teac

her e

duca

tion.

Chri

stin

e Ca

rolin

e Jä

hnig

. (p.2

4) A

sses

sing

pro

cedu

ral K

now

ledg

e of

Ger

man

Bus

ines

s Stu

dent

s.M

icha

el B

ots.

(p.24

) Aut

hent

icity

and

por

tfolio

ass

essm

ent.

Wednesday 29 august


4

Mai

n co

nfer

ence

room

Oly

mpi

a Bo

urse

Ba

llroo

m

Brea

kout

room

1

Euro

Brea

kout

room

1

Ster

ling

Brea

kout

room

1

Dol

lar

Brea

kout

room

1

Yen

9h-10h30

Invi

ted

Sym

posi

um (p

.27)

Shar

on K

linke

nber

gIn

vite

d Sy

mpo

sium

SIG

-DT,

Th

eme:

Test

-enh

ance

d le

arni

ng m

eets

lear

ning

an

alyt

ics.

Sym

posi

um (p

.28)

Lies

je C

oert

jens

Stud

ent a

ppro

ache

s to

lear

ning

and

thei

r pe

rfor

man

ce: E

vide

nce

from

prim

ary

and

high

er

educ

atio

n.

10h30-11h

Coffe

cor

ner

11h00-12h30

Sym

posi

um (p

.30)

Sue

Blox

ham

Asse

ssm

ent:

safe

in o

ur

hand

s?

Jan

Ard

ies

(p.33

)D

evel

opm

ent o

f an

Asse

ssm

ent T

echn

olog

ical

Li

tera

cy.

E. P

anad

ero

(p.35

)Fr

iend

ship

in th

e re

liabi

lity

of n

on-r

ubric

an

d ru

bric

-bas

ed p

eer

asse

ssm

ent.

Caro

lin R

amst

eck

and

Ann

ette

Fru

ehw

acht

(p.37

)A

Feed

back

Inte

rven

tion

Theo

ry

Pers

pect

ive

on E

xter

nal S

choo

l pe

rfor

man

ce D

ata

Use

in

Ger

man

Sec

onda

ry C

lass

room

s.

Sasc

ha W

üste

nber

g (p

.38)

Com

plex

Pro

blem

Sol

ving

in

Educ

atio

nal C

onte

xts –

Asse

ssm

ent,

Mea

sure

men

t Inv

aria

nce,

and

Co

nstr

uct V

alid

ity.

Carm

en T

omas

(p.33

)In

tegr

atio

n of

e-a

sses

smen

t pr

oces

ses f

or la

rge-

scal

e in

stitu

tiona

l dev

elop

men

t.

Robb

ert S

mit

(p.35

)Ru

bric

s as a

link

be

twee

n st

anda

rds a

nd

form

ativ

e as

sess

men

t.

Chan

g Zh

u (p

.37)

Prov

idin

g fo

rmat

ive

feed

back

to

stud

ents

via

em

ails

and

fe

edba

ck st

rate

gies

rela

ted

to

stud

ent m

etac

ogni

tion.

Gor

don

Joug

hin

(p.39

)M

ultip

le p

ersp

ectiv

es o

n a

viva

voc

e as

fina

l ass

essm

ent i

n ve

terin

ary

scie

nce.

Kris

Buy

se (p

.34)

Effec

ts o

f str

ong

port

folio

tr

igge

rs o

f heu

ristic

in

stru

men

ts a

nd li

nked

su

mm

ativ

e/fo

rmat

ive

asse

ssm

ent o

n w

ritin

g,

writ

ers a

nd tu

tors

.

And

ers

Jons

son

(p.36

)Sc

orin

g ru

bric

s for

st

uden

t lea

rnin

g.

Sara

h M

alon

e (p

.38)

Com

pute

r bas

ed a

sses

smen

t of

driv

ing

com

pete

nce

invo

lvin

g di

ffere

nt re

spon

se m

odes

Céci

lle R

aves

loot

(p.40

)Q

ualit

y im

prov

emen

t of s

umm

ativ

e ra

diol

ogic

al sk

ills a

sses

smen

t with

di

gita

l mul

ti-sl

ice

imag

es in

med

ical

ed

ucat

ion.

12h30-14h

Rest

aura

nt

Mid

twon

Gril

l

14h- 15h30

Keyn

ote

(p.40

)M

enuc

ha B

irenb

aum

15h30-16h

Coffe

cor

ner

16h-17h30

Sym

posi

um (p

.41)

Mar

gare

t Pric

eRe

conc

eptu

alis

ing

Feed

back

in

Hig

her E

duca

tion.

Rutg

er K

appe

(p.43

)Ac

adem

ic a

sses

smen

t cen

ters

an

d st

uden

t com

pete

ncie

s: Re

liabi

lity,

cons

truc

t, co

nseq

uent

ial a

nd p

redi

ctiv

e va

lidity

.

Julie

Bro

snan

(p.45

)Ea

rly li

tera

cyas

sess

men

t and

Inte

rven

tion.

Mar

y H

ill (p

.47)

The

impa

ct o

f a d

edic

ated

as

sess

men

t cou

rse

on

deve

lopi

ng te

ache

r edu

catio

n st

uden

ts’ a

sses

smen

t cap

abili

ty.

Jam

es P

elle

grin

o (p

.49)

A Va

lidity

Fra

mew

ork

for

Asse

ssm

ents

Des

igne

d to

Sup

port

In

stru

ctio

n.

Vale

ntin

a G

rion

(p.44

)Sc

hool

qua

lity

eval

uatio

n:

stud

ents

’ poi

nts o

f vie

w

Mar

y Sh

eard

(p.46

)Eff

ects

of S

elf-

Pace

d Le

arni

ng o

n Ac

hiev

emen

t in

Sent

ence

Leve

l Wor

k in

Ye

ar 5

.

Katr

ien

Stru

yven

(p.48

)Th

e el

ectr

onic

por

tfolio

as a

n in

stru

men

t for

the

deve

lopm

ent

of te

ache

r com

pete

nces

in p

re-

serv

ice

teac

hing

pra

ctic

e.

Susa

n O

rr (p

.49)

Ded

ucin

g ‘le

arni

ng o

utco

mes

in

actio

n’ fr

om le

ctur

ers’

asse

ssm

ent

talk

.

Carm

en V

idal

Rod

eiro

(p.45

)D

o di

ffere

nt a

sses

smen

t ro

utes

(lin

ear v

s. m

odul

ar)

prep

are

stud

ents

, in

the

sam

e w

ay, f

or fu

rthe

r stu

dy?

Dav

id H

idal

go (p

.47)

Prac

tical

pro

blem

s in

the

asse

ssm

ent o

f tex

t co

mpr

ehen

sion

.

Lisa

Sm

ith

(p.48

)St

uden

t and

Teac

her E

duca

tor

Perc

eptio

ns o

f Bec

omin

g As

sess

men

t-Ca

pabl

e Te

ache

rs.

Fran

ces

Edw

ards

and

Bev

erle

y Co

oper

(p.50

)D

evel

opin

g ‘a

sses

smen

t cap

able

’ be

ginn

ing

seco

ndar

y te

ache

rs in

N

ew Z

eala

nd.

Thursday 30 august


5

Mai

n co

nfer

ence

room

Oly

mpi

a Bo

urse

Ba

llroo

m

Brea

kout

room

1

Euro

Brea

kout

room

1

Ster

ling

Brea

kout

room

1

Dol

lar

Brea

kout

room

1

Yen

9h-10h30

Invi

ted

Sym

posi

um (p

.27)

Shar

on K

linke

nber

gIn

vite

d Sy

mpo

sium

SIG

-DT,

Th

eme:

Test

-enh

ance

d le

arni

ng m

eets

lear

ning

an

alyt

ics.

Sym

posi

um (p

.28)

Lies

je C

oert

jens

Stud

ent a

ppro

ache

s to

lear

ning

and

thei

r pe

rfor

man

ce: E

vide

nce

from

prim

ary

and

high

er

educ

atio

n.

10h30-11h

Coffe

cor

ner

11h00-12h30

Sym

posi

um (p

.30)

Sue

Blox

ham

Asse

ssm

ent:

safe

in o

ur

hand

s?

Jan

Ard

ies

(p.33

)D

evel

opm

ent o

f an

Asse

ssm

ent T

echn

olog

ical

Li

tera

cy.

E. P

anad

ero

(p.35

)Fr

iend

ship

in th

e re

liabi

lity

of n

on-r

ubric

an

d ru

bric

-bas

ed p

eer

asse

ssm

ent.

Caro

lin R

amst

eck

and

Ann

ette

Fru

ehw

acht

(p.37

)A

Feed

back

Inte

rven

tion

Theo

ry

Pers

pect

ive

on E

xter

nal S

choo

l pe

rfor

man

ce D

ata

Use

in

Ger

man

Sec

onda

ry C

lass

room

s.

Sasc

ha W

üste

nber

g (p

.38)

Com

plex

Pro

blem

Sol

ving

in

Educ

atio

nal C

onte

xts –

Asse

ssm

ent,

Mea

sure

men

t Inv

aria

nce,

and

Co

nstr

uct V

alid

ity.

Carm

en T

omas

(p.33

)In

tegr

atio

n of

e-a

sses

smen

t pr

oces

ses f

or la

rge-

scal

e in

stitu

tiona

l dev

elop

men

t.

Robb

ert S

mit

(p.35

)Ru

bric

s as a

link

be

twee

n st

anda

rds a

nd

form

ativ

e as

sess

men

t.

Chan

g Zh

u (p

.37)

Prov

idin

g fo

rmat

ive

feed

back

to

stud

ents

via

em

ails

and

fe

edba

ck st

rate

gies

rela

ted

to

stud

ent m

etac

ogni

tion.

Gor

don

Joug

hin

(p.39

)M

ultip

le p

ersp

ectiv

es o

n a

viva

voc

e as

fina

l ass

essm

ent i

n ve

terin

ary

scie

nce.

Kris

Buy

se (p

.34)

Effec

ts o

f str

ong

port

folio

tr

igge

rs o

f heu

ristic

in

stru

men

ts a

nd li

nked

su

mm

ativ

e/fo

rmat

ive

asse

ssm

ent o

n w

ritin

g,

writ

ers a

nd tu

tors

.

And

ers

Jons

son

(p.36

)Sc

orin

g ru

bric

s for

st

uden

t lea

rnin

g.

Sara

h M

alon

e (p

.38)

Com

pute

r bas

ed a

sses

smen

t of

driv

ing

com

pete

nce

invo

lvin

g di

ffere

nt re

spon

se m

odes

Céci

lle R

aves

loot

(p.40

)Q

ualit

y im

prov

emen

t of s

umm

ativ

e ra

diol

ogic

al sk

ills a

sses

smen

t with

di

gita

l mul

ti-sl

ice

imag

es in

med

ical

ed

ucat

ion.

12h30-14h

Rest

aura

nt

Mid

twon

Gril

l

14h- 15h30

Keyn

ote

(p.40

)M

enuc

ha B

irenb

aum

15h30-16h

Coffe

cor

ner

16h-17h30

Sym

posi

um (p

.41)

Mar

gare

t Pric

eRe

conc

eptu

alis

ing

Feed

back

in

Hig

her E

duca

tion.

Rutg

er K

appe

(p.43

)Ac

adem

ic a

sses

smen

t cen

ters

an

d st

uden

t com

pete

ncie

s: Re

liabi

lity,

cons

truc

t, co

nseq

uent

ial a

nd p

redi

ctiv

e va

lidity

.

Julie

Bro

snan

(p.45

)Ea

rly li

tera

cyas

sess

men

t and

Inte

rven

tion.

Mar

y H

ill (p

.47)

The

impa

ct o

f a d

edic

ated

as

sess

men

t cou

rse

on

deve

lopi

ng te

ache

r edu

catio

n st

uden

ts’ a

sses

smen

t cap

abili

ty.

Jam

es P

elle

grin

o (p

.49)

A Va

lidity

Fra

mew

ork

for

Asse

ssm

ents

Des

igne

d to

Sup

port

In

stru

ctio

n.

Vale

ntin

a G

rion

(p.44

)Sc

hool

qua

lity

eval

uatio

n:

stud

ents

’ poi

nts o

f vie

w

Mar

y Sh

eard

(p.46

)Eff

ects

of S

elf-

Pace

d Le

arni

ng o

n Ac

hiev

emen

t in

Sent

ence

Leve

l Wor

k in

Ye

ar 5

.

Katr

ien

Stru

yven

(p.48

)Th

e el

ectr

onic

por

tfolio

as a

n in

stru

men

t for

the

deve

lopm

ent

of te

ache

r com

pete

nces

in p

re-

serv

ice

teac

hing

pra

ctic

e.

Susa

n O

rr (p

.49)

Ded

ucin

g ‘le

arni

ng o

utco

mes

in

actio

n’ fr

om le

ctur

ers’

asse

ssm

ent

talk

.

Carm

en V

idal

Rod

eiro

(p.45

)D

o di

ffere

nt a

sses

smen

t ro

utes

(lin

ear v

s. m

odul

ar)

prep

are

stud

ents

, in

the

sam

e w

ay, f

or fu

rthe

r stu

dy?

Dav

id H

idal

go (p

.47)

Prac

tical

pro

blem

s in

the

asse

ssm

ent o

f tex

t co

mpr

ehen

sion

.

Lisa

Sm

ith

(p.48

)St

uden

t and

Teac

her E

duca

tor

Perc

eptio

ns o

f Bec

omin

g As

sess

men

t-Ca

pabl

e Te

ache

rs.

Fran

ces

Edw

ards

and

Bev

erle

y Co

oper

(p.50

)D

evel

opin

g ‘a

sses

smen

t cap

able

’ be

ginn

ing

seco

ndar

y te

ache

rs in

N

ew Z

eala

nd.

Mai

n co

nfer

ence

room

Oly

mpi

a Bo

urse

Bal

lroo

mBr

eako

ut ro

om 1

Eu

roBr

eako

ut ro

om 1

St

erlin

gBr

eako

ut ro

om 1

D

olla

rBr

eako

ut ro

om 1

Ye

n9h-10h30

Invi

ted

Sym

posi

um (p

.51)

Liz

McD

owel

lAs

sess

men

t for

lear

ning

: va

ryin

g pe

rspe

ctiv

es w

ith a

fo

cus o

n le

arne

rs.

Sym

posi

um (p

.53)

Mar

ieke

van

der

Sch

aaf

Teac

her a

nd st

uden

t ro

les d

urin

g fe

edba

ck in

in

tera

ctio

n.

10h30-11h

Coffe

cor

ner

11h00-12h30

Gyo

ngyv

er M

olna

r (p.5

6)In

nova

tive

Asse

ssm

ent

Tech

nolo

gies

in E

duca

tiona

l G

ames

Des

igne

d fo

r In

tegr

atin

g As

sess

men

t int

o Te

achi

ng.

Gill

Elli

ot (p

.58)

How

mul

tiple

per

spec

tives

co

mpl

icat

e co

mpa

rabi

lity.

Jan-

Will

em S

trijb

os (p

.59)

Dev

elop

ing

a co

ding

sche

me

for a

naly

sing

pee

r fee

dbac

k m

essa

ges.

Dia

na B

aas

(p.61

)Pe

rcep

tions

of t

he p

urpo

ses

of p

ortfo

lio a

sses

smen

t in

prim

ary

educ

atio

n.

Ale

x Ko

zulin

(p.62

)D

ynam

ic co

gniti

ve a

sses

smen

t of

youn

g ad

ults

.

Lász

ló H

ülbe

r (p.5

6)Co

mpa

ring

diffe

rent

item

-ty

pes i

n pa

per-

base

d an

d co

mpu

teriz

ed e

nviro

nmen

t.

Ildik

ó Ba

lázs

i (p.5

8)Fi

rst r

esul

ts o

f Hun

gary

’s fu

ll co

hort

long

itudi

nally

co

nnec

ted

asse

ssm

ent.

Dan

iel G

arci

a-A

lmiñ

ana

(p.59

)Pr

ompt

Fee

dbac

k by

an

asse

ssm

ent i

nstr

umen

t (E

valc

omix

) in

an E

ngin

eerin

g D

egre

e

Mic

hiel

Vel

dhui

s (p

.61)

Prim

ary

scho

ol te

ache

rs’ u

se

of c

lass

room

ass

essm

ent

in D

utch

mat

hem

atic

s ed

ucat

ion.

Vict

oria

Que

sada

-Ser

ra (p

.62)

How

do

univ

ersi

ty te

ache

rs a

sses

s th

eir s

tude

nts?

Und

erst

and

asse

ssm

ent f

rom

thei

r ow

n vi

ewpo

int.

Gen

evie

ve G

auth

ier (

p.57)

Asse

ssm

ent P

lann

ing

for

Inte

ract

ive

Case

-bas

ed

Lear

ning

Sce

nario

.

Kris

tine

Høe

g Ka

rlse

n (p

.60)

Feed

back

mat

ters

whe

n st

uden

ts a

re co

nsid

erin

g w

hat t

ype

of le

arni

ng

activ

ities

to e

ngag

e in

- ex

plor

ing

the

rela

tions

hip

betw

een

feed

back

-typ

es a

nd

stud

ents

lear

ning

act

iviti

es in

hi

gher

edu

catio

n

Birg

it D

en O

uter

(p.63

)As

sess

men

t lite

racy

in u

nive

rsity

st

uden

ts: w

hat i

s it a

nd h

ow is

it

deve

lope

d?

12h30-14h

Rest

aura

nt

Mid

twon

Gril

l

14h- 16h

Deb

ate

Stan

dard

-bas

ed n

atio

nal

test

ing.

+ Cl

osin

g

16h-End

Coffe

cor

ner

Friday 31 august


6

Wednesday 29 augustKeynote

Up or Out and the Joy of Lifelong Learning. Joseph Kessels

Raising performance standards, evidence based instruction, compulsory professional develo-pment and strict accountability are recurrent topics in the debates on school improvement and the enhancement of educational quality. National and international rankings of student achieve-ments act as signposts for success and failure. Policy makers, school managers and heads of de-partments negotiate about financial means and learning outcomes. Assessment and evaluation play a crucial role in these debates. From a critical perspective assessment is probably the most political of all educational processes.

In view of the need for lifelong participation in learning and development it is relevant to explo-re the influence of external achievement pressures on students and their teachers. Is it a matter of up or out? Is an appreciative way of assessment helpful in establishing self-esteem, professional identity and a joy of lifelong learning?

Symposium

Symposium Chair: David Boud

There has been considerable interest on feedback in higher education worldwide prompted in part by consistent negative reports of surveys of student opinion that have been critical of uni-versities, and in part by a desire by scholars to make sense of a phenomenon that has been driven more by convention than by research. Useful review papers by Hattie and Timperley (2007) and by Shute (2008) have drawn attention to the existence of considerably more research on feedback than has been commonly recognised.

Conventional views of feedback are being found wanting. There is a need for new perspectives on feedback to be developed to recognise the diverse demands being placed on it. The sympo-sium papers argue for new and multi-faceted perspectives on feedback. The first provides a criti-que of the ways in which the idea of feedback has been appropriated from engineering without realising the full import of the idea and without taking account of the very different nature of students from engineering systems. It suggests that a new conception of feedback is needed for the educational context and sketches some of the key features of such a conception. The second paper also recognises the importance of the human dimension in feedback and argues that trust needs to be introduced as a key idea in how feedback is used and considered. Feedback is not only about the transmission of information from one person to another, but about the trustwor-

Symposium Discussant: Dai Hounsell

11h-12h30: Main conference room - Olympia Bourse Ballroom



7

Paper 1.- Reframing feedback: new perspectives, new dimensions. David Boud

The paper argues that feedback needs to be reconceptualised to focus on desired educational outcomes beyond the immediate task. Feedback is a metaphor appropriated by education from its origins in other disciplines. This paper suggests that, in translation, this idea (Feedback Mark 1) has been misunderstood and misused in educational contexts, and that a vital feature from its engineering usage has been lost in its use in education. It argues that it is not a simple matter of reinstating the missing concept, useful though that might be. A more thorough reappraisal is needed in recognition of learners as human agents who cannot be regarded as inanimate systems susceptible to the feedback intentions of others.

The paper explores then, what a new conception of feedback, native to the field of education, could look like that was (Feedback Mark 2). It integrates recent scholarship of feedback and as-sessment that emphasises the importance of developing in students a sense of agency regarding their own learning, dialogue between givers and receivers of feedback in clarifying and apprecia-ting standards and criteria, educational sustainability of feedback so that it contributes to lear-ning beyond the immediate task, the use of multiple providers and receivers of feedback, and the design of educational programs to embed feedback activities utilising these features as a normal part of teaching and learning activities (from Boud, Carless, Hounsell, Nicol, etc.). It also incorpora-tes from Feedback Mark 1, the feature most commonly absent in education: the detection of the effect of interventions on subsequent production. In doing so it positions the effective use of fee-dback not just as a learning device, but as an intended learning outcome for the curriculum. The paper concludes by discussing the implications of such a reframing of feedback for educational practice and for research on assessment.

David CarlessTrust has long been recognized as an important issue in various fields, including economics,

political science, sociology and organizational theory. Trust is insufficiently discussed in relation to teaching, learning and assessment in higher education; and when it is mentioned it is often a minor sub-theme rather than a topic of extended discussion. The aim of this paper which is con-ceptual in nature is to analyze some key dimensions of trust in relation to feedback.

Many common barriers in the feedback process relate to ineffective communication or lack of trust. These include challenges to affect and the emotions; different perceptions of feedback amongst students and teachers; and some student misgivings about the role of peers in provi-ding feedback. For feedback to be effective lecturers and students need to enter into open and dialogic relationships of trust.

thiness of both provider and recipient of that information. Students cannot be expected to act on information if they do not believe that in doing so will be in their best interests. The third paper recognises the key role of student agency in feedback and links feedback with self-assessment undertaken by learners as a means for utilising information from teachers.

Paper 2.- Trust, distrust and their role in the feedback process.


8

The paper analyzes two dimensions of trust which carry relevance for the feedback process: communication trust and competence trust. Communication trust includes: empathetically lis-tening and responding sensitively to the thoughts of others; communicating frequently so that trust is developed through repeated interactions; and confidence that a feedback-provider has the integrity to behave fairly and ethically.

Competence trust or ‘trust of capability’ involves investing faith that others have the ability and the willingness to provide useful feedback on performance. Feedback from individuals with rele-vant expertise is more likely to be accepted and acted upon.

Two strategies are discussed which carry potential to develop trust: the further development of assessment literacy and enhanced communication processes. The development of assessment literacy amongst both teachers and students can strengthen competence trust. Enhanced trans-parency and communication about feedback may assist students in developing a deeper unders-tanding of the feedback process.

Paul OrsmondHigh achieving students in comparison to non-high achieving students are effective in using

tutor feedback in their learning due to their ability to self-assess. Therefore, encouraging students to learn from tutor feedback does not necessarily require a direct focus on the feedback provided. Rather it involves developing student’s ability at self-assessment. Tutors need to provide scaffol-ding for students so they recognise the importance of self-assessment in their learning. One way of doing this is to provide course infrastructure when designing curriculum, learning activities and assignments so that both tutors and students are able to focus on the role of ‘self’ in learning. A framework for achieving this is the GOALS process.

The GOALS process is student focused considering learning from a student perspective in the now: G = what outcomes do I need to Grasp. These can be self generated outcomes or tutor outco-mes which as translated by the student so that they understand them. O = how can I Orientate to ‘self’ perhaps through asking questions such ‘What am I trying to achieve?’ A = what Actions do I need to take to achieve my outcomes. This could be seen in terms of ‘who do I need to speak to’ or ‘how can I most effectively use the tutor provided exemplars’. The GOALS process also considers learning for the future. L = what type of Learning evaluation do I need to make. Asking questions such as ‘How have I done?’ or ‘How am I different now compared to when I started?’ ‘What has changed?’ Finally S = what Strategies do I need to develop in order to successfully move on. Stu-dents need to think about the experience of doing an assignment, perhaps in terms of how they can reduce the ‘error’ in their approach.

Paper 3.- Learning from tutor feedback: a role for self-assessment.


9

From an educational effectiveness perspective, research in the analyses of international compa-rative data has been limited as it neglects to take macroeconomic and developmental differences of countries into account. Further, methodological challenges of cross-sectional studies have as yet prevented the evaluation of international comparative data from a longitudinal effectiveness perspective. Therefore public league-table reporting of results yields biased judgments of suc-cessful and less successful educational systems.

In this paper methodological approaches commonly applied in Educational Effectiveness Research (EER) are used to comparatively analyze international large-scale achievement data. Drawing on data from the Progress in Reading Literacy Study (PIRLS) 2001 and 2006 it adjusts for non-malleable factors, creating the basis for “fair comparisons”. Further, it investigates how effectively educational systems grow, i.e. change, in their performance. Specifically, a methodolo-gical approach known from school effectiveness research is applied to investigate changes at the country-level with different cohorts of students.

Data from PIRLS trend countries is analyzed, applying a three-level hierarchical linear model that nests students in schools, in countries and includes a cohort variable at school level. Effective-ness measures of achievement status in 2006 and of changes from 2001 to 2006 are investigated.

Results suggest that there are countries which outperform their expected outcomes (status and change) as well as countries which stay below it, changing the picture of “high” and “low” perfor-ming educational systems, when contextual conditions and prior performances are taken into account.

The study enhances methodological developments of educational effectiveness research in cross-national assessments. Its results are relevant when international large-scale assessments are used to compare educational systems on the basis of their performance and yield a more reliable foundation for looking at policies, practices, structures, and ideologies that have helped countries to be successful, i.e. effective.

Paper A New Look at the Evaluation of Sociological Theories in International Large Scale Educational Assessments.

Daniel Caro • Andrés Sandoval-HernandezThis paper uses a mixed approach of theory generating and theory testing to evaluate theo-

retical models of economic, cultural and social capital. The data stem from two large scale in-ternational assessments: PIRLS 2006 and PISA 2009. In line with the mixed approach employed, the method is a combination of exploratory factor analysis (EFA) and structural equation models (SEM) known as exploratory structural equation modeling (ESEM). The main advantage of using this method is that an EFA measurement model with rotations can be used in a SEM, instead of a

Paper How effective are educational systems? A value-added approach to study trends in PIRLS.

Jenny Lenkeit14h-15h30: Breakout room 1 Euro

14h-15h30: Breakout room 1 Euro


10

PaperRevisioning National Assessment in New Zealand.

Alison Gilmore • Jeffrey Smith • Charles DarrThe National Monitoring Study of Student Achievement (NMSSA) is a new programme in New

Zealand designed to assess the school progress of primary school children at year 4 (roughly 8 years old) and year 8 (roughly 12 years old). It is a low-stakes, light-sampling approach that inclu-des both paper and pencil measures, performance-based tasks, interviews, and affective as well as cognitive assessments. The programme replaces the internationally recognised National Edu-cation Monitoring Project, which ran from 1995 to 2009. The programme is currently completing a developmental year, and will begin formal assessment activities in late 2012.

NMSSA is conducted jointly by the Educational Assessment Research Unit (EARU) of the Uni-versity of Otago, and the New Zealand Council for Educational Research (NZCER). The conceptual framework for NMSSA uses innovative approaches to national assessment that extend traditional approaches in terms of what is assessed, how it is assessed, and how it will be communicated. One of the goals of the new programme will be to move beyond simply assessing achievement to uncovering the underlying reasons for that achievement and for why it might differ in different segments of the population. The EARU/NZCER collaborative team has been working on the deve-lopment of this new programme for over a year as of the time of this proposal.

The purpose of the proposed presentation is to lay out the ideas that have been developed for the programme, with a special emphasis on new approaches to assessment, and new thinking on extending the notion of assessment to include why achievement occurs in addition to how much has occurred. We will discuss the theoretical framework of the programme in addition to presen-ting the results of an extensive series of pilot studies.

Paper Effects of Keyboarding Skills on Performance on Computer-Based L2 Writing Tests.

Khaled BarkaouiA major concern with computer-based (CB) tests of second-language (L2) writing is that perfor-

mance on such tests may be influenced by examinee keyboarding skills. From a cognitive pers-pective (e.g., Kellogg, 1996), poor keyboarding skills may force examinees to focus their attention and cognitive resources on motor activities (i.e., typing) and, consequently, other processes and

14h-15h30: Breakout room 1 Sterling

confirmatory factor analysis (CFA) measurement model. When using the EFA we have the advan-tage of being able to relax the zero loading restriction, allowing variables to load on different fac-tors simultaneously. The SEM then provides model fit indices, yields measurement invariance tests across countries, and corrects factor scores for measurement error. The results provide statistical criteria to evaluate the fit of theories with the empirical data (i.e., more reliably and validly). The paper contributes to the discussion on the advantages and disadvantages of using these methods when evaluating sociological theories in the context of international large scale assessments.



11

aspects of writing (e.g., organization, revising) might be left unattended to, which can lead to poor text quality and lower test scores. Such effects might be more pronounced for L2 writers (Wolfe & Manalo, 2005). Few studies, however, have examined the impact of keyboarding skills on exami-nees’ performance on CB L2 writing tests.

This study investigated the impact of keyboarding skills on examinees’ test scores and writing processes. Each of 100 examinees (25 examinees x 2 L2 proficiency levels x 2 keyboarding skill levels) responded to two TOEFL-iBT writing tasks (independent and integrated) on the computer. A keystroke logging program recorded all writing sessions. Each essay was then rated by two in-dependent TOEFL-iBT raters.

Keystroke data were coded in terms of students’ pausing behaviors. Pauses provide observable and measureable cues as to where and how long writers halt in order to plan and revise their texts, whereas the linguistic output provides cues as to the nature of the hidden revision and planning processes (Matsuhashi, 1981). For example, long pauses are likely to be associated with macro-planning processes, whereas shorter pauses reflect microplanning processes (e.g., lexical choices). Pause were coded in terms of frequency, duration, type (formulation or revision pause), and tem-poral and linguistic location. Test scores and pause data were then statistically compared across keyboarding skill groups, L2 proficiency levels and task types. Findings and their implications for CB L2 writing assessment and research will be discussed.

Paper Developing national standards for the assessment of writing in com-pulsory education.

Astrid Birgitte Eggen • Synnøve MatreThe presentation reports from a project aiming at carrying out in-depth research on specified

assessment norms for writing, and their subsequent effects on the quality of pupils’ writing and teachers’ rating. The project builds on a social semiotic construct of writing that yields construct and content validity as well as principles for organizing standards and a multidimensional scoring instrument.

The project is designed in two stages. The objective of stage one is, in collaboration with 8 schools across Norway, to define which levels of writing competency should reasonably be ex-pected at specified grades as a basis for setting explicit standards. The objectives of stage two (intervention) are, in collaboration with 16 schools, a) to investigate what effects a shared set of explicit standards for educational writing, across nationally distributed local learning ecologies, may have on the quality of teacher assessment, in particular for rater agreement, and b) to inves-tigate what effects an integration of shared explicit standards in teaching may have on the quality of pupils’ writing. In stage two qualitative and quantitative methods are combined, and the study is designed as a repeated- measurements field study with six partially overlapping sub-studies.

Methods and findings will be presented from stage one of the project. Qualitative data were collected from a number of schools, using semi-structured small-group and individual interviews with teachers, supplemented by pupils’ texts. Findings, i.e. levels of writing competency expected at grades 5 and 8 of the compulsory school as to various dimensions of a written text (commu-nication, content, composition, use of language, encoding and presentation), will be presented.



12

Paper Examining Task Equivalence in L2 Writing Tests: A Mixed MethodsApproach.

Khaled BarkaouiExamining and establishing task equivalence in second-language (L2) writing tests that employ

various tasks across test occasions and/or offer examinees task choice is an essential step to en-sure valid score interpretations and fair decisions across examinees, tasks and testing occasions. This study illustrates how the question of task equivalence in L2 writing assessment could be em-pirically examined using both qualitative and quantitative data. We collected 46 essays on two ar-gumentative tasks, that are assumed to be equivalent in terms of their difficulty, but that differ in terms of orientation (private vs. public), from 27 Arab L2 university students. The essays were then rated by six of the students’ L2 writing teachers. Building on Connor and Mbaye’s (2002) Writing Competence Model, we developed a framework for analyzing the essays in terms of their gram-matical, discourse, sociolinguistic and strategic features. We then compared essay features and scores across writing tasks and examined the relationships between essay features and scores. The public task was more difficult and led to relatively higher inter-rater agreement, more diver-se and sophisticated vocabulary, more formal features, and better argument quality. The private task resulted in higher scores, longer essays and more informal features. The texts also differed in terms of their rhetorical structure across tasks. Finally, raters varied significantly in terms of the scores they assigned and their perceptions of the essays. The paper discusses the main findings of the study as well as the data collection and analyses strategies used in the study, particularly the operationalization of the Connor-Mbaye Writing Competence Model, and how they could be used in research on task equivalence and variability in examinee and rater performance in L2 wri-ting assessment.

Paper Spanish as Foreign Language Teachers’ conceptions ofassessment: preliminary results from an internet inquiry.

Gavin Brown • Ana RemesalWe present preliminary results of a study on the conceptions of assessment of ELE-Teachers

(ELE = Spanish as Foreign Language) from around the world. An online questionnaire was sent out via an online professional community with over 7500 members (Comunidad Todoele). The questionnaire got a response ratio of 57%; hence, the sample is 4275 respondents, representing different professional contexts: (1) teaching Spanish as Foreign Language in compulsory school, (2) as an extra-academic activity for children and teenagers; (3) in adult education, and (4) within in-company training.

The questionnaire of conceptions of assessment was designed following a four dimensional and bi-polar model (Remesal, 2006; Remesal, 2011). Conceptions are understood as the subjecti-

14h-15h30: Breakout room 1 Dollar


The results should initiate a discussion about 1) the relationship between integration of explicit standards of writing performance and the effects on pupils’ writing, and 2) their utility value as a basis for implementing nationwide tests of writing.


13

ve sum of individual beliefs, which, in turn, are assumptions about objects and phenomena that people take as true (often without intellectual contrast) (Green, 1971; Pajares, 1992). This model of conceptions differentiates among beliefs about the effects of assessment on teaching, on lear-ning, on certifying knowledge, and on account giving. For each of these dimensions, the beliefs about assessment can aggregate either as an overall conception of ‘assessment as an instrument for positive regulation of the teaching and learning process’ or a conception that ‘assessment is an instrument for societal control’.

Exploratory factor analysis of the data point to significant differences between the four teaching contexts: in-school, extra-school, adult education, and in-company. It also reveals differences in conceptions of assessment in relation to the teachers’ teaching practice (traditional top-down lan-guage teaching versus communicative bottom-up model), and their own experience as assessed subjects during their own school experience. Thus, the results highlight a need for a specific edu-cation for assessment that helps the teachers develop assessment competence if assessment for learning is the goal.

Paper Assessment in Brazilian higher education: Validation of a Portuguese version of the Students’ Conceptions of Assessment (SCoA-VI)inventory.

Gavin Brown • Daniel A. S. Matos • Walter L. Leite • Sergio D. CirinoStudents’ conceptions of assessment have been investigated with a self-report survey inven-

tory that aggregates 33 items into 8 factors forming four major conceptions (i.e., assessment is for improvement, assessment brings social/personal benefit, assessment is irrelevant, and as-sessment is a reflection of external attributes). A Portuguese version of the Students’ Conceptions of Assessment (SCoA) inventory was administered in two Brazilian universities (N=756). Confirma-tory factor analysis used the mean and variance-adjusted weighted least squares estimator and good fit to the original model, developed in New Zealand with secondary students, was found. Students also indicated the practices they associated with assessment; four types of assessment practices were found: informal student-controlled; informal teacher-controlled; formal student-controlled; and formal teacher-controlled. The students emphasised conventional, formal evalua-tion methods controlled by the teacher. Even supposedly student-controlled practices (i.e., self and peer-assessments) were perceived as teacher-controlled practices. As a whole, the university students agreed most with assessment being for improvement, and second for accountability. The definitions of assessment were meaningfully predicted by key factors in the SCoA inventory in a well-fitting structural equation model. Together these relations suggest that students think: “Formal assessments are not enjoyable, but they are not bad because they help me improve my learning. If assessment is informal and controlled by the teacher, it’s good for class dynamics and climate of the classroom, and I do not ignore this type of assessment activity.” The level of educa-tion, language, and country did not interfere with how the SCoA items bundled together to make latent traits indicating that the same latent constructs exist in two samples from quite different populations. The structural pathways between SCoA factors and definitions of assessment are un-



14

Paper Conceptions of Classroom Assessment: Findings from UsingQ-Methodology.

Wei Shin LeongThis paper details the pilot use of Q-methodology (Q) for understanding the different possible

conceptions of classroom assessment among Singaporean teachers. This work builds on an earlier study involving the phenomenographic method of interviewing teachers on their perspectives of assessment. To my knowledge, Q has not been used in education assessment research even though it is known to be a useful methodology for studying different teachers’ perspectives. My interest is currently in the tacit decision-making process in wide ranging social interplays among teachers in the practical situations that Q researchers tend to prefer to investigate. The pilot the-refore serves at least two main broad purposes: firstly, to challenge myself to try new methodo-logy in my current stage of the research journey into teachers’ classroom assessment and invite feedback from experts en route. Secondly, to study the process and results from the use of Q and what I have learnt that could have possible implications for my future research work and for other researchers, teachers and policy-makers. A key question is whether the Q is helpful to investiga-te the possible multiplicity and significant relationships of conceptions of classroom assessment within a specific context of Singaporean teachers’ complex classroom decision-making environ-ment, whereby teachers are empowered to make significant decisions on curriculum, pedago-gy and assessment. The different clusters of conceptions from the Q-factor analysis, in particular, have revealed underlying bases of common and different views of classroom assessment and implications for practices. This suggests opportunities for consensus-building that will be useful for teachers, policy-makers, professional learning developers and researchers.

Paper Teacher Communication of Evaluation Results: Impact on Student Engagement in School.

Carolina Carvalho • Joseph Conboy • Maria Odete ValenteStudent engagement is a factor in students’ school enjoyment and success; it may influence de-

cisions to stay in school and provide an opportunity for at-risk students to achieve social equality. However, some evaluation practices may act to reduce student engagement. Secondary educa-tion in Portugal is marked by high-stakes testing that can influence acceptance in higher edu-cation programs. In this study, we listened to what some student voices say about evaluation practices in use in secondary schools in Portugal. Tenth- and 11th- year students (N=105) in two secondary schools in the Lisbon area, provided structured reflective narratives about the impact

derstood to be ecologically rational. Students express belief patterns that are consistent with the ways assessment is used in their educational contexts.

The authors acknowledge financial support to the first author from Coordenação de Aper-feiçoamento de Pessoal de Nível Superior (CAPES).


14h-15h30: Breakout room 1 Yen


15

(both as students and persons) of types of communication teachers use when reporting evalua-tion information. An initial reading of the student responses indicates that students have few po-sitive episodes to relate about teacher assessment. Students clearly object to unjust interpreta-tion of their academic effort and condemn abusive and humiliating actions on the part of some teachers, as well as teacher sarcasm and irony when presenting evaluation results. According to the students, a poor grade can increase engagement if it is accompanied by teacher support and belief in the student’s ability; it can lead to disaffection if the low grade is unexpected, perceived as unjust, or does not correspond to the effort exerted. Students indicate that they prefer clear information from teachers that points the way as to how to improve. They point out that a sum-mative classification does not provide this information. Poor human relations on the part of some teachers may also lead to student disaffection.

PaperThinking about Fairness in Classroom Assessment.

Robin TierneyFairness is an enduring ideal and a perpetual concern in educational assessment. Until recently,

most research relating to fairness focused on standardized tests or large-scale assessments. Grea-ter interest in classroom assessment has raised questions about fairness, particularly because of the personal nature of some methods, the interaction involved in the use of assessment for learning, and the need to serve diverse students. While classroom assessment was once conside-red low-stakes in comparison to standardized testing, it is now widely recognized that teachers’ judgements can have profound consequences for students. Fairness issues are often mentioned in classroom assessment literature, but clear definitions of the concept are less common. Inter-pretations of fairness vary considerably, and they are often reported by group membership (i.e., teachers, students), which suggests that fairness is a matter of perspective. However, by linking different perspectives, fairness may be more productively considered as a single concept with multiple, interrelated facets. This research brings the perspectives of highly experienced and spe-cialized teachers into a scholarly dialogue, and draws on their practical wisdom to reconceptualize fairness for the dynamics of classroom assessment. Practical wisdom (phronesis) is essentially a fluid network that uses different forms of knowledge, including theoretical, practical and ethical knowledge, to determine the best course of action in particular circumstances. The six participants in this multi-case study were purposefully selected based on years of teaching experience, type of degree, and level of specialization. While each offered a unique perspective on fairness, patterns were evident across their recommendations. Six aspects of fairness, involving transparency, stu-dents’ opportunities, equitable treatment, critical reflection, and the classroom environment, are discussed in relation to a model of assessment for learning. The goal in presenting this work is to facilitate thinking and generate discussion within the educational community about the meaning of fairness in classroom assessment.



16

Roundtable Accuracy in Self-assessment: Debating the Merit of Accuracy when Students Evaluate their Own Work.

Gavin Brown • Heidi L. AndradeThis roundtable will present divergent views on the importance of accuracy in student self-eva-

luation. The dominant view in Assessment for Learning policies and interpretations of classroom assessment is a pedagogical interpretation of assessment arguing that student self-assessment of work products and processes is useful for raising academic performance. This view draws on self-regulation of learning theories of how students improve their own work against socially-defined goals. Students can learn to set targets, evaluate progress relative to target criteria, and conse-quently improve the quality of their learning outcomes. Hence, there has been a strong adoption of student self-assessment in classroom practices, with little or no concern for the validity of stu-dents’ self-assessments. .

PaperThe emotional dimension of feedback.

Carol EvansThis study focuses on the emotional dimension of feedback in order to better understand how

to enhance feedback practice within higher education institutions (HEIs) and specifically addres-ses the question of how to enhance student engagement in and with feedback. An interpretative phenomenological analysis approach was adopted (Smith, 2003) whereby the subjective expe-riences of student teachers (STs), to inform programme development, was the focus. The two year study involved 34 postgraduate STs in two UK HEIs. The findings of the first year study were used to inform development of the second year feedback intervention as part of an action research project funded by the Higher Education Academy, UK. Data gathered via group and individual interviews, as well as email communications were analysed using content-analysis procedures to identify over-arching themes (Krippendorff, 2004). The emotional dimension of feedback was one of the emergent meta themes, identified as underpinning how students accessed and made use of feedback opportunities. Implications for policy and practice from the study are twofold. Firstly, a number of elements in the design of assessment and feedback were identified that could be enhanced to reduce student anxiety, particularly in relation to feed-forward holistic assessment design elements. Secondly, in developing sustainable feedback practice, the importance of stu-dent self-regulation of feedback is essential (Boud, 2000; Hounsell, 2007) and management of the affective dimension of feedback is imperative in this respect. It was evident that “savvy fee-dback seekers” were more able to manage the emotional dimension of feedback compared to their peers. In developing assessment practice as an integral element of teaching, it is important that HEIs address the emotional dimension of feedback explicitly with students in order to sup-port them in developing strategies to maximise their ability to self-regulate their learning in order to engage in and with feedback.




17

Introduction of quality assurance elements into the assessmentpolicy of generic graduate attributes of university using the course-embedded assessment methods.

However, the reliability of testing is considered a sine qua non for the validity of assessment interpretations, and research into psychological processes underlying the human ability to self-evaluate work raises serious doubts about the quality of students’ judgments. It has been shown that novices (i.e., students) tend to lack sufficient knowledge by which to evaluate their own work appropriately and, when possessing that knowledge, they are likely not to make use of it. Addi-tionally, psychological safety factors indicate that self-assessment can be compromised by inter-personal relations present in classroom environments. Furthermore, since progress in many edu-cational domains is relatively ill-defined, it is difficult for learners, let alone instructors, to validly evaluate progress or status.

While self-assessment is important, there is still a debate as to how accurate such evaluations need to be. While research has shown how accurate student self-assessment can be, little has been done to investigate whether it is necessary that the self-assessments are accurate. There are different perspectives on how to move forward with implementing self-assessment in classrooms. This roundtable will examine quality issues in this important classroom assessment practice.

Roundtable

Patrick LaiDeveloping graduate generic competencies is one of the major responsibilities of universities.

Literature on generic graduate attributes focuses on studies of Barrie in (1) understanding of ge-neric attributes by university teachers; (2) revision of university’s statement of generic graduate attributes and (3) development of a conceptual framework for the teaching and learning of ge-neric graduate attributes. There is NO study that discusses valid and reliable methods to assess generic graduate attributes on an institutional basis.The significance and originality of this pa-per extends literature on generic graduate attributes by exploring and evaluating the usefulness of course-embedded assessment (CEA) as an institutional learning outcomes measure, with two possible directions of using CEA for institutional outcomes assessment. On-going observations and interviews with 11 university teachers from 7 departments throughout the project and a final round of post-project interviews with 9 key players of the project team for data triangulation were done. Interviews prompts for the post-project interviews focus on (1) recommendations for using course-embedded assessment (CEA) as a tool for institutional assessment; (2) strategies to enable valid, reliable data collection and (3) pros and cons of the two suggested possible directions of using CEA for institutional outcomes assessment were included.Findings of the observations and post-meeting interviews address the issues of (1) whether it is possible to use programme selec-ted artifacts to assess outcomes at institutional level and the changes that need to be made to en-sure consistency; (2) the extent to have common criteria and standards with respect to outcomes across programmes and a preferred way to make the process reliable and valid. Finally two pos-sible directions with respect to an institutional framework of outcomes assessment is discussed.



18

Roundtable The relationship between past and present in end of term tests in Flemish history education.

Karel Van Nieuwenhuyse • Kaat Wils •Geraldine Clarebout • Lieven VerschaffelEmpirical research on the way in which a school subject is assessed, gives a privileged entrance

into teaching practices. It enables to go beyond official curricular goals and teachers’ statements about the subject, and gives insight into what teachers consider in the end to be important goals to achieve and competences. Our presentation will be based on the analysis of a corpus of 200 written end of term tests in history for the 11th and 12th grade in schools within the Flemish Com-munity (the Dutch-speaking, northern part of Belgium). As such, it connects to the first strand of the conference, on summative assessment.

The analysis will focus on the position of the present and the relationship between past and pre-sent in the end of term tests. Teachers in Flanders are explicitly encouraged to integrate current issues in their lessons and to ‘actualize’ their teaching on the past (which, according to the official curricula, in the 11th and 12th grade, deals with the period from 1789 to the present). At the same time, however, they are expected to stimulate students to think historically, and hence to pay at-

RoundtableWhat Types of Assessment Contexts Support Self-Regulated Learning?

Huiyong Seah-Tay • K.H. Koh • K. TanThe propose paper examines the relationship between self-regulated learning (SRL) and

authentic assessments using a mixed method design, with sequentially quantitative and qualita-tive phases, to answer the research questions: 1) Was there a difference in students’ use of SRL in a conventional paper-and-pen task and an authentic assessment task? 2) How was SRL facilitated or hindered by real life setting, school setting, student control and the student’s ability to self-assess? To cross-validate the findings, the study was replicated two years later with another group of par-ticipants at the same site. Quantitative data on SRL was gathered using a self-report questionnaire I had designed based on Zimmerman’s (2002) cyclical Forethought, Performance and Self-reflec-tion SRL model. Participants, comprising 405 secondary four students in the first study and 300 in the second, were assigned two tasks that tapped on the same cognitive skills, except that one was a paper-and-pen task while the other was an authentic assessment task. Comparisons of the SRL scores between the two tasks, using paired sample t-tests, showed a small statistically significant difference in favour of the authentic assessment task and none in the second. Further analysis at the subscale and phase level added a more nuanced understanding of the relationship between the two phenomena. These inferences were supported by interview data from 19 participants: 6 from the first study and 13 from the second. Their comments corroborated the literature review that real life setting, student control and ability to self-assess facilitated students’ SRL. Furthermo-re, the study provided a clearer picture of how these factors are linked. These findings suggest reframing our view on formative and summative classroom assessments, especially with regards to authentic assessments.




19

Roundtable Understanding of Assessment by University Lecturers: how this links to theories.

Maddalena Taras • Mark DaviesThis paper reports work in progress on a series of studies that examines lecturers’ understan-

dings of aspects of assessment in a UK HEI. These aspects include summative and formative as-sessment processes, products and functions and the relationship between them, and how fee-dback and learner involvement relate to them.

Empirical data were collected from 50 education lecturers, 50 science lecturers and 12 academic developers through questionnaires and also through semi-structured interviews. The data were examined qualitatively and quantitatively to provide an overview of understandings of aspects of assessment, feedback and learner involvement.

This paper focuses on the differences and similarities in understandings between the education and science lecturers in order to ascertain if greater access by education lecturers to pedagogic theories increases the understandings of different aspects of assessment.

Due to the inconsistencies in the education literature, the research posits that education lec-turers will reflect the representations in the literature and therefore have an incomplete and un-harmonious understanding of assessment terms and how they relate and interrelate. As concerns science lecturers, it is posited that without the conflicting theories that education lecturers are required to engage with, it will result in a high degree of coherence in their reasoning and unders-tandings.

Initial evaluation of the results indicates that there is a difference in understanding between education and science, although the differences are not as great as were originally predicted. By exploring and evaluating the results, this paper provides an overview of the thinking and unders-tanding of lecturers and examines how this will impact on practice and therefore students.

It further permits the participants to also question their own understandings and positions on some of these issues and therefore consider how they can refine their assessment procedures and the support of students’ learning.

tention to the specific historical context of past phenomena. This tension holds a major challenge for teachers. Previous research among (future) history teachers suggests moreover that they are more geared towards the present in their general beliefs on history education than in their actual teaching practices.

The in-depth analysis of the end of term tests will shed light on the extent to which teachers pay attention to the present in their concrete educational practice, and how they approach the pre-sent and its relation to the past. Based on this analysis, intervention strategies will be proposed to achieve a (more) conscious and balanced handling of this tension, to enhance students’ learning of history.



20

RoundtableFormative Assessment using Collective Portfolios.

Núria de SalvadorIn the context of competence-oriented learning, virtual portfolios using wikis can be appropria-

te tools to carry out formative assessment. However, motivating pupils to produce in a wiki increases the assessment load of the teacher

and decreases his or her control over pupils’ productions. The leitmotiv of this field research is to reflect on means to facilitate assessment in this context,

and offer technical and methodological solutions to it. This proposal arises from a real need for a course in teaching English in secondary and VE education, where wikis are used as virtual portfo-lios for the last two school years.

Roundtable High School Student’s Portfolios in Science for All: Indicators of the Quality of their Teachers’ Work.

Amira Rom • Michal NachshonThe Curriculum for Science for All is based on processes which make connections between tea-

ching, learning and evaluation. Its goal is to instill science technology literacy into high school students who do not specialize in science. The main tool used in the evaluation process is that of the portfolio which includes representative products of the students’ work, as evidence of the learning process, and the students’ achievements and progress. The products are a reflection of their scientific literacy and include complex individual and group assignments which are evalua-ted by means of a competency rubric.

The teacher’s role in school is to find alternative means of problem solving for students, in ac-cordance with his more pronounced area of intelligence. Everything points to the need for a wide ranging change in outlook on learning achievement evaluation, encompassing changes in tea-ching strategies, the teachers’ role and in the teacher-student relationship.

Evaluation of the teachers’ work is an important component of the educational process. When attention is devoted to the evaluation process and its influence on teaching and learning, a practi-ce evolves which positively effects and improves the quality of these process. .There are two main purposes in evaluating a teacher’s work: improved performance and establishment of accounta-bility.

The study goal was to investigate students’ portfolios as indicators of the quality of their tea-chers work.

The findings point out the strong and weak points of the teachers and suggest guide lines for future teacher training and preparation in order to best implement alternative evaluation in the field.

As a result of these findings the research suggests ways to improve the quality of work of the teachers and the use of the portfolio as a system wide evaluation tool in Science for All classrooms specifically and in secondary schools in general.




21

Masahiro Arimoto • Janet Looney

Roundtable Exploring undergraduate students’ informal, formative peerassessment networks.

Rita HeadingtonPeer assessment incorporates peer-marking, peer-correction, peer-rating, peer-feedback, peer-

review and peer-appraisal (Topping 1998). Updating Topping’s study, Gielen et al’s (2011) inven-tory demonstrates increased application and diversity of peer assessment practice integral to un-dergraduate students’ academic studies. Gielen et al (2011) do not discuss students’ transferral of taught peer assessment methods or skills to informal contexts, yet it is known (eg HEFCE, 2007; Ryan 2008, Sutton and Taylor 2011) that students use online and face-to-face social networks, of peers, family members and work colleagues, to undertake informal, formative peer assessment of coursework and assignments.

Nicol (2010) sees peer assessment as supplementary to tutor-student interaction, not separate from it. Additionally, Orsmond and Merry (2009) identify differences in high and low achieving stu-

This paper presents case studies of small rural schools in Akita, one of Japan’s top-scoring pre-fectures in the 2007 National Assessment of Academic Ability. The authors focus on the concept of “formative assessment” for lesson study. A framework on formative assessment to support “lear-ning to learn” is extended to include Wenger’s concept of the “communities of practice”, The paper describes mutual engagement, joint enterprise and the development of shared repertoires within these communities.

The results of the case studies showed that (1) teachers use jargon based on the Japanese “ki-zuki” to refer to embedded assessment, rather than more formal assessment criteria or methodo-logy. They also try to incorporate the Japanese concept of “neriai” (polish, elaborate, work out or knead lesson well) into the core pedagogical culture. “Neriai” means a discussion phase with the whole class discussion phase for structured problem solving. It is the core of teaching through problem solving. It is accompanied with teachers’ double loop learning and support for learning circles based on social capital. (2) a “soft middle” level allows for modifications to take place in policy making; teacher-initiated approaches combines elements from different subjects, so that “bottom-up” approaches undergo a transformation throughout the process of implementation. “Learning to learn” it is discussed more extensively. The municipalities have built system leader-ship around this concept to open the door to other experts and expand frontiers (e.g. removing the barriers to welfare). They are formally accountable to the national bureaucracy, and also an intimate and very real accountability to their colleagues. Teachers are involved in their students’ lives outside of school and are in constant communication with the parents, they are accountable to the parents and grandparents in a unique way.

Linking a framework of formative assessment and evaluation for “learning to learn” to learning circles for transformational school leadership.

Roundtable




22

Roundtable Students’ underperformance due to pseudo understanding of math-ematical concepts: What can be improved in school assessment?

Ho Yi XianPseudo-understanding describes a phenomenon where students show evidence in apparent

understanding of particular concepts, but have yet achieve thorough grasp of the topic or sub-ject matter. While pseudo-understanding allows students to perform well at summative tests and exams, it poses a critical problem when they progress to higher and deeper levels of learning. Gaps in their prior understanding of topics, concepts and ideas hinder students’ learning and in-ternalizing of new knowledge. A qualitative study of four pre-university students, who had scored distinctions in their previous national examinations but did not do well in their college tests and exams, revealed that one of the key obstacles for the construction of new knowledge was stu-dents’ pseudo-understanding of mathematical concepts from their lower levels of study. Despite students giving written answers and working which could be considered acceptable and correct, clinical interviews with the students revealed gaps and misunderstandings in several concepts which they had previously learnt in secondary school. This may be a result of assessment prac-tices during their secondary school years. It was found that students were unable to self-assess, they were not habitual in reviewing their work, and they do not question, explain, defend, reason and challenge their understanding of mathematical concepts. Assessment practices which may have contributed to students developing pseudo-understanding include excessive repetition of tasks which resemble those given in summative tests and rushing for students to attempt exam typed questions in multiple timed pen and paper testing, even before students could make sense and abstract ideas from manipulations. Shift of focus to formative assessment and assessment for learning can encourage students to reflect on their learning and challenge conceptions that they formulated which minimizes pseudo-understanding of mathematical concepts and facilitate knowledge construction at higher levels of learning.

dents’ commensurate abilities to interpret and make use of feedback. As these studies highlight tutors’ roles in managing group composition to facilitate effective peer assessment within social constructivist frameworks, is informal, student-led peer assessment inevitably a deficit model of practice?

Set in a UK university, and forming the first part of a longitudinal study of students’ interpreta-tion and use of formative and summative feedback, this study explores undergraduate initial tea-cher education students’ informal, formative peer assessment networks, focusing on ‘composition of assessment groups’ and ‘interaction between peers’ (Gielen et al 2011). Using social network analysis (Knoke and Yang 2008; Carrington and Scott 2011), it investigates who students choose as informal peer assessors within their personal networks, how the formative assessment usually takes place (eg online, face-to-face) and what students hope to gain by choosing to work with particular individuals.

This paper provides outcomes, identifies and discusses issues emerging from the study and cri-tiques the methodology. It considers the appropriateness of students’ informal networks for peer assessment and the support they require to use networks effectively.



23

Roundtable Will digitally aiding the creation of a project plan createtransfer?

Steven Nijhuis

Roundtable UITZIGT integrated assessment of competences on outflow level in teacher education.

Kristien Carnel • Katelijne De Munter • Ilse Loockx • Ann Adriaenssen • Van Heddegem Ilse • Bieke Quaghebeur

In competence learning we need assessments in which knowledge, skills and attitudes are inte-grated. In competence oriented education we see a growing need of a more integrated approach to final assessment, if possible in authentic learning contexts.

UITZIGT (Uitstroomcompetenties voor zij-instromers geïntegreerd toetsen) is funded by the School of Education (SoE) within the K.U.Leuven association 2010-2012. The project aims to de-velop a toolkit for the qualitative assessment of competences at the end of the training program based on teaching internships.

Based on a concise literature research we investigated which assessment tools are best suited for assessing certain basic competencies and competence clusters referring to the professional profile of a teacher. This lead us to the development of a blueprint UITZIGT, an assessment battery to test competences of oncoming teachers as they are ready to finish.

Teacher education is increasingly confronted with students that apply for short track programs. APEL procedures and more competence based curriculum designs increase the need for proper

To support students in higher education in creating a project plan for their project based lear-ning assignments, a Digital Coach was created and successfully implemented. The Digital Coach promotes reflection of the students in preparing the project plan.

Project plan benefitted greatly from the students use of the coach, as could be expected [1, 2]. In this workshop the context of the coach (project based learning, design research, use) is descri-bed and the coach is used.

Focal point of the discussion is whether the Digital Coach supports learning, specifically the importance of creating a sound project plan. It cannot be concluded that the coach supports transfer. Which gives rise to the question how to research the effect of transfer and even whether this research will prove to be useful [3].

[1] Paans, W. (2011). Accuracy of nursing diagnoses: Knowledge, knowledge sources and reaso-ning skills. (PHD, Katholieke Universiteit Leuven).

[2] Paans, W., Sermeus, W., Nieweg, R., & van der Schans, C. (2010). Determinants of the accuracy of nursing diagnoses: Influence of ready knowledge, knowledge sources, disposition toward criti-cal thinking, and reasoning skills. Journal of Professional Nursing, 26(4), 232-241.

[3] Eiriksdottir, E., & Catrambone, R. (2011). Procedural instructions, principles, and examples: How to structure instructions for procedural tasks to enhance performance, learning, and transfer. Human Factors, 53(6), 749-770.




24

RoundtableAssessing procedural Knowledge of German Business Students.

Christine Caroline JähnigAssessment of student competencies is a highly relevant topic for research, practitioners and

policy makers. Unfortunately the number of available competence tests for German universities is rather limited. In the field of Business Administration Studies only a small number of standar-dized tests is available that primarily focus on assessing declarative knowledge. Considering the competence orientation of the Bologna Process and the complex demands young business pro-fessionals face when leaving the university, a need for action-oriented complex assessment is postulated. This contribution introduces an innovative test type to assess procedural knowledge of undergraduate business students. Procedural knowledge is the knowledge exercised in the accomplishment of a task, and thus closely related to the action-oriented competencies. Unlike declarative knowledge, it cannot be easily articulated by the student and is often described as tacit. The test combines elements of Situational Judgement Tests (SJTs) with approaches of tacit knowledge measurement. The challenge of meeting demands of curricular and criterion validity when constructing a situational test for higher education will be discussed.

instruments for final assessment that better suits alternative learning pathways typical for these target groups.

Funded by a project of School of Education of the Catholic University of Leuven 3 institutes for Teacher Education (KHLeuven, CVOVIVO Kortrijk, Lessius Mechelen) developed an assessment ba-ttery that responds to these intentions.

The UITZIGT project aims to assess the outflow competences based on alternative assessment procedures. Based on a concise literature research we investigated which assessment tools are best suited for assessing basic competencies and competence clusters referring to the profes-sional profile of a teacher. This lead us to the development of a blueprint UITZIGT, an assessment battery to test competences of oncoming teachers as they are ready to finish their study program.

This paper provides outcomes, identifies and discusses issues emerging from the study and cri-tiques the methodology. It considers the appropriateness of students’ informal networks for peer assessment and the support they require to use networks effectively.

RoundtableAuthenticity and portfolio assessment.

Michael Bots • Jolanda de PutterWithin Dutch higher education many teacher education programs rely on the use of electronic

portfolios for assessing professional development and competence. Despite the growing research in this area, many practitioners and researchers are still looking for the best way to combine stan-dardization for optimal reliability with an open structure that encourages authenticity.

This paper attempts to contribute to the discussion by presenting the results and implications




25

Godfrey Pell • Richard Fuller • (Matt Homer)Institutions have conflicting pressures of quality, feasibility and cost in the assessment of stu-

dents. Students desire a demonstrably fair assessment process which is, ideally, completed by all in a single academic year. This most typically takes the format of assessment of the whole student cohort, followed by a period of remediation and retesting for those who underperform.

However, resit/borderline candidates are not a homogenous group and considerable difficulty persists in accurately identifying and profiling these ‘at risk’ students. Emergent work also reveals that traditional models of test-remediate-retest may not lead to sustained long term improve-ment in performance for these students, raising major implications in assessment policy for edu-cational institutions (Pell et al 2009; Hauer et al 2009).

This interactive workshop will overview existing literature in this area, and encourage parti-cipants to recognise the limitations of their own assessment strategies and methodologies for underperforming students vis-a-vis competent students, and the implications for policy. The de-velopment of sequential test methodologies, based on regression towards the mean phenome-na (Bland & Altman, 1994) will explore a way of meeting this challenge. Participants will explore theoretical and practical applications of sequential testing methodology, and gain confidence in practical approaches to managing cost-benefit analyses and quality issues in assessment policy.

A strong focus of the workshop will be the discussion of the limitations of existing research in this field, and encouragement of participants to share their own good practice and opportunities for collaborative research in this important field. This workshop has particular significance for all disciplines using performance based/criterion based assessment approaches.

of a research study in the Netherlands. Using a combination of qualitative research methods the project has investigated the current use of portfolio assessment in the field of teacher training and the perceptions of students, assessors and supervisors on (the authenticity of) different portfolio structures and procedures. The results provide a deeper understanding in the way students reflect and anticipate on the assessment procedure, and how assessors arrive at a conclusion by using all kinds of (and more or less relevant) information.

In the round table session we will present the results and discuss some of the preliminary con-clusions on guidelines that will improve the authenticity of portfolio assessment. Together with the audience we will discuss the implications of these guidelines for the reliability, efficiency and transparency of portfolio assessment.

The challenge of meeting the assessment needs of both competent and underperforming students within existing educational structures: Introducing sequential testing? Lessons for policy and practice.

Workshop 1



26

Workshop 3 A theory of improvement for teachers’ professional development in Assessment for Learning.

N. van Benthum • J. Van der Steen • J. Gulikers • F. De Jong • M. MulderVocational education in The Netherlands is engaged in the introduction of competence-based

education (CBE) (De Bruijn & Leeman, 2011). In this reform, it’s needed to focus on Assessment for Learning (AfL) which first priority is to serve the purpose of promoting students’ learning (William, 2011). However, teachers are usually not trained in designing AfL in CBE (Popam, 2001; Birenbaum, 2003). Therefore, it’s crucial to distinguish features which stimulate teachers’ professional develo-

Workshop 2EvalCOMIX: A web service for u-feedback and u-feedforward.

Eduardo García • Mª Soledad Ibarra • Gregorio RodríguezThe EVALfor research group has developed a series of projects based upon the principles of

learning oriented assessment (Carless, 2006) and sustainable assessment (Boud, 2000). One of the main products of this research is the web service EvalCOMIX (http://evalcomix.uca.es), specifically designed and developed to aid teachers and students in implementing participative assessment activities. EvalCOMIX achieves this using strategies such as self-assessment, peerassessment and co-assessment.

This workshop provides the necessary information and resources for the participants to use the EvalCOMIX web service. Once completed, participants will have acquired the following skills:

•Design of assessment instruments (checklists, rating scales, semantic differentials and rubrics).•Design assessment instruments that promote feedback and feedforward.•Use of instruments for teacher assessment processes, self-assessment, peerassessment and

co-assessment.•Analyse results from the assessment tasks undertaken.

The workshop has been designed as an essentially practical exercise in which participants de-sign and develop their own assessment instruments for use in Learning Management System (LMS). To this end, participants will initially be presented with the basic characteristics of the Eval-COMIX web service (design, management, import and export of assessment instruments). The first part will then conclude with a practical exercise in which each participant will develop at least one rating scale and one simple rubric.

After that, a presentation will be given on ways to use the designed assessment instruments, in order to perform teacher assessment, self-assessment, peerassessment and co-assessment.

Finally, participants will be presented with the possibilities EvalCOMIX offers for analysis of re-sults obtained from an assessment process, by means of analysis of the different graphs presented by EvalCOMIX.

All the instruments offered by EvalCOMIX can be used not only on a personal computer, but also on other more current devices such as smartphones or tablets. EvalCOMIX thus transcends the li-mits of e-assessment and brings forward the beginning of ubiquitous assessment (u-assessment).




27

pment in AfL in a theory of improvement (ToI). A ToI is a description of how features of a pro-fessionalization program stimulate teachers’ learning and is often absent in teacher professional development (Van Veen, Zwart, Meirink en Verloop, 2010).

In this study, a ToI is designed to professionalize teachers in AfL. The ToI is based on a literatures-tudy and is applied in a program to improve teachers’ and managers’ assessment practice in Dutch Agricultural Vocation Education and Training (AVET).

Characteristics of the ToI are: 1. Conceptions on assessment which influence how teachers deal with assessment in practice (Ho, Watkins & Kelly, 2001; Segers & Tillema, 2011). 2. Assessment practice because classroom assessment practice is the change context (Smith, 2011) and should be challenged by giving feedback on actual practice (Gardner et al, 2010). Conceptions are cha-llenged in teachers’ assessment practice and puts the teachers in a constant 3. inquiry to impro-ve practice (Birenbaum et al, 2011). 4. Congruent professional development in which teachers learn to apply AfL in classrooms and managers to apply AfL to stimulate teachers’ learning in as-sessment. This characteristics of the ToI lead to the most suitable approach for this professional development in a Professional Learning Community (PLC). A PLC is closely related to the concept of collaborative learning communities.

Study is running and preliminary results show potential of this ToI to improve assessment prac-tice to AfL.

Thursday 30 august

Research on the development of expertise performance (Ericsson, 2006, pp. 683–703) has shown that people improve their performance considerably if they regularly do specific exerci-ses that are adjusted to their ability level and include immediate feedback. In this digital age and using computer adaptive practice (Klinkenberg, 2011) it is possible to monitor the ability of stu-dents while they are in their learning process. To deliver immediate feedback it is necessary to know what kind of errors an individual makes. This calls for analysing error types, indicating the relative frequency of these errors and provide feedback. In this invited symposium we will explore where test-enhanced learning meets learning analytics.

Sharon Klinkenberg • Marthe Straatemeier • Han van der MaasIn this paper we present a model for computerized adaptive practice and monitoring. This mo-

InvitedSymposium

Invited Symposium SIG-DT, Theme: Test-enhanced learning meets learning analytics.

Symposium Chair: Sharon Klinkenberg Symposium Discussant: Silvester Draaijer9h-10h30: Main conference room - Olympia Bourse Ballroom

Paper 1.- Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation.


28

Silvester DraaijerThe need to increase student success rates, especially in the first years of higher education, is

becoming stronger and stronger. Online quizzing in the first years of undergraduate education is regarded as one possible way to improve time-on-task, student engagement and retrieval ability. Several studies are published that address the incorporation of online quizzing in higher edu-cation. These studies show mixed results with regard to study success. For example, in general, student opinions towards the use of online quizzes is positive, however the effect on time-on-task is just modest and the correlations with success rate vary from no correlation to significant corre-lations with modest effects. The studies incorporate various course set-ups and qualitative and quantitative analyses methods and take a different number of variables into account. These could be the cause of the different findings. In this paper, an overview of several studies is presented and discussed. Online quizzing will be related to the ideas of deliberate practice and possibilities of data-mining and feedback (learning analytics) to improve the possible generation of data on which students and educators can act on and to increase the impact of online quizzing.

Symposium Student approaches to learning and their performance: Evidence from primary and higher education.

Due to its clear educational relevance, the link between student approaches to learning and their study outcomes has been frequently investigated in the context of higher education (e.g., Vermunt, 2005; Ward & Walker, 2008). Yet, the results on this issue do not speak in one voice: mixed results have been reported. The manner in which academic performance in operationalized could play a key role in explaining these mixed findings. By using different measures for academic per-formance, the first study contributes significantly in verifying this hypothesis.

The relationship between student approaches to learning and performance has been often looked at during the period of higher education. The question arises however whether students’ approaches have already predictive power earlier on in the educational curriculum. Focussing on the learning conceptions on students in primary education, the second study asks the whether

Symposium Chair: Katrien Struyven Symposium Discussant: Liesje Coertjens

del is used in the Maths Garden, a web-based monitoring system, which includes a challenging web environment for children to practice arithmetic. Using a new item response model based on the Elo (1978) rating system and an explicit scoring rule, estimates of the ability of persons and the difficulty of items are updated with every answered item, allowing for on the fly item calibra-tion. In the scoring rule both accuracy and response time are accounted for. Items are sampled with a mean success probability of .75, making the tasks challenging yet not too difficult. In a pe-riod of ten months our sample of 3648 children completed over 3.5 million arithmetic problems. The children completed about 33% of these problems outside school hours. Results show better measurement precision, high validity and reliability, high pupil satisfaction, and many interesting options for monitoring progress, diagnosing errors and analyzing development.

Paper 2.- Using online quizzing and quiz results to improve study success.



29

Mikaël De Clercq • Benoit Galand • Mariane FrenayThe aim of this study was to investigate the impact of motivational and cognitive processes on

several achievement outcomes. Academic achievement was assessed through final percentage and three tasks assessing three different levels of knowledge (acquisition of knowledge, unders-tanding of principles and application of concepts). The relation between achievement and self-efficacy beliefs, goal orientation, learning strategies and self-regulation was investigated while controlling for student’s previous performance. The participants were 217 students from the en-gineering faculty at the Université catholique de Louvain in Belgium. The students were assessed at the beginning of their third year of study. Data were analyzed through stepwise regression. The results mainly highlighted that final percentage is essentially modulated by motivational fac-tors (mastery goal), whereas the performance to specific tasks was related with cognitive factors (deep processing and surface processing). Moreover, the results also revealed that the impact of self-efficacy beliefs on achievement outcomes vanished when student’s previous performance are controlled for. The implications of our results on the understanding of the relation between learning process and academic achievement have been discussed.

Eric Robbers • Peter van Petegem • Vincent Donche • Sven De MaeyerGiven the fundamental character of Learning Conceptions and the fact that learning concep-

tions develop overtime into conceptions that are found in secondary education pupils and in higher education students (Marton, Dall’Alba & Beaty, 1993), insight into learning conceptions of young pupils is relevant especially in correlation with learning results.

The objective of this research is to determine whether there is correlation between learning conceptions of pupils in grade eight of primary education and their learning results. The sample contained 649 twelve years old pupils in grade eight of primary education spread among 16 pri-mary schools in the south east of the Netherlands. Learning conceptions were measured with the revalidated learning conceptions questionnaire (authors). Learning results were measured using the primary school education final progression test. Given the hierarchic structure in the data multilevel analysis was used.

Research findings show that children in grade eight of primary education who scored high on the scales of negative attribution and external regulation of the learning conception questionnai-

the predictive validity holds at this age as well. This would offer possibilities for early remedial activities.

The link between of student approaches and performance is not only of substantive but also of methodological interest. In longitudinal studies on the change in learning strategies, this me-chanism generates a biased follow-up sample. Such selective sample could in turn jeopardize the correctness of the estimates of change in learning strategies over time. The third study investi-gates whether the missingness can be considered not completely at random, using the data of a three-year study in a Flemish University College.

Paper 1.- Learning processes in higher education: Comparing the impact of cognitive and motivational variables on specific and global measures of achievement.

Paper 2- Correlation between learning conceptions and final school results in primary education.


30

Liesje Coertjens • Vincent Donche • Sven De Maeyer • Peter Van PetegemLongitudinal studies on the change in learning strategies during higher education are on a rise

(Vanthournout et al., 2011). As is common with this design, there is an amount of attrition (Sha-dish, Cook, & Campbell, 2002). By applying listwise or pairwise deletion prior to analysis of change over time (e.g., Zeegers, 2001), most longitudinal studies take the assumption that this missing-ness is completely at random (MCAR) (Enders, 2010). The students who did not participate in all waves are thus considered a random subset of those persisting in higher education and in the research. Yet, given the literature on the link between learning strategies and learning outcomes achievement (e.g., Furnham, Monsen, & Ahmetoglu, 2009; Yip, 2009), this assumption is questio-nable. Recent methodological advances allow for estimating whether missingness can be seen as not at random (MNAR) or non-ignorable (Enders, 2011). This study innovates in applying this to a longitudinal study on the growth in students’ approaches to learning during higher education. One cohort of Flemish University College was followed during their course in higher education and were administered the ILS-SV (Donche & Van Petegem, 2008) at each of the three years.

Preliminary results indicate that for the relating and structuring, self-regulation, external regu-lation scales the MCAR assumption does not hold. The missingness can be assumed dependent on the previously observed scores for these scale (e.g. at the first wave). Moreover, for the lack of regulation, self-regulation and external regulation the missingness is estimated being not at ran-dom. Implications of these results are discussed.

Symposium Assessment: safe in our hands?

Higher education assessment is a massive, global, edifice involving students, academics and administrators as millions of assignments, exams, performances and other tasks are assessed and graded annually. At the heart of this unstoppable behemoth are institutions’ and subject discipli-nes’ academic standards. Confidence in these standards has been questioned in recent years in se-veral in several HE systems (Brown, 2010; Coates, 2010, Vanderslice, 2004) prompted by a growing diversity and complexity in higher education (Woolf & Cooper, 1999). There are claims of grade inflation and lack of parity of standards as Universities and academics largely retain the right to set and maintain their own standards. So, is assessment safe in our hands?

Symposium Chair: Sue Bloxham Symposium Discussant: Clair Hughes

Paper 3- Is missingness in longitudinal research during higher education dependent on student’s learning strategies? A Flemish case study.

re, scored significant lower on the final school test for primary education. For children in the last year of primary education, learning results are significant lower when they have a high score on the variables negative attribution and regulation by teachers. The fact that learning conceptions change after transition from primary to secondary education (Klatter, 2003), means that the sti-mulation of pupils learning conceptions in the direction of the correct form of regulation, using more profound processing activities possibly leads to better learning results.



31

Sue BloxhamThis paper considers academic standards in assessment; an imperative at a time of political and

reputational pressure in the standards arena. Assumptions underpinning standards are discussed drawing on contrasting theoretical perspectives; the techno-rational perspective which under-pins quality assurance of assessment and the hermeneutic account which emphasises the inter-pretive nature of academic judgement.

Neither epistemological approach provides a satisfactory basis for grading policy which is illus-trated by the contrast between the assumptions underpinning quality assurance of standards (re-liable, consistent, transparent) and the everyday practice of grading (shifting, socially constructed and tacitly held standards). Yet millions of assessments are graded unproblematically each year. The paper will consider how these contrasting perspectives are negotiated by lecturers in the act of grading work as a basis for developing a policy agenda for academic standards in use.

It draws on a mixed-method study of faculty from four disciplines and 3 universities. ‘Think aloud’ protocols and interviews were utilised to access both actual and espoused data regarding grading practices. The findings illuminate how lecturers are implicitly combining different epis-temologies in negotiating work practices. Whilst recognising that much of what they do is inter-nalised and tacit, assessors gain a sense of security about their marking judgements from various quality assurance documents and procedures. The act of creating and discussing, for example, assessment criteria provides a clear opportunity for detailed dialogue about standards. There is a strong influence of ‘techno-rational’ artefacts on interpretive practice as many quality assurance processes for assessment become normative.

However, the findings continue to reinforce the local and co-constructed nature of standards albeit within a framework of dialogue about explicit statements of standards . The implications for developing appropriate policy regarding assessment standards, consensus, accountability and information for students will be discussed.

This symposium brings together three papers which investigate this question from theoretical, policy and practice perspectives. The first paper examines the failure of existing theory to provi-de a workable model for standards in use. It explores the lack of alignment between empirical evidence regarding academic judgement processes and the assessment information provided to students. The second paper provides an insight into approaches to policy implementation regar-ding standards. It explores features of a range of nationally-funded projects designed to enhance the understanding and use of standards in Australia’s evolving regulatory environment. The final paper explores professional judgement at the practice level. It reports on an empirical study re-garding how different members of the academic profession come to know academic standards as operationalised in their grading of student work.

The symposium debate will draw on commonalities in the papers regarding ways forward for policy in rebuilding confidence in assessment, for example using communities, dialogue and exemplars. It will provide the opportunity to debate how standards might be more effectively understood and used in the academy at different levels and with different audiences.

Paper 1.- “You can see the quality in front of your eyes”: developing a policy agenda for academic standards in use.


32

Margaret PriceThe problems of shifting from norm-referenced assessment to criterion-referenced assessment

have been identified by several scholars in recent years. These important critiques touch on a number of areas, but neglect a key question about how assessors learn to accomplish what Shay calls a ‘socially situated interpretive act’ (2004). Criterion-referenced assessment is premised on clear standards applied consistently by all assessors in a community. However the limitations in making standards explicit and the consequent problems in sharing common understanding of standards means that the criterion-referenced assessment remains ‘faulty’ especially where the-re are multiple assessors working together. New assessors or incomers to a local community of markers have to quickly try to assimilate the standards in use in order to belong. Research which does exist tends to focus on salaried, full-time academics. This overlooks the heterogeneity of the academic labour force in HE, and the substantial contribution made by contract (hourly-paid) lec-turers, particularly in applied disciplines such as Business and Management.

This study explores how nine newcomers to a UK Business School - including salaried and con-tract staff - attempt to understand local assessment practice. We use a situated learning lens to analyse their diary entries and interviews about their experiences of learning to mark. Drawing

Clair HughesAn intensification of interest in the quality of teaching and learning in higher education has led

to international trends in defining and articulating the standards of academic achievement (Harris 2009). While commonalities can be observed among the ways the standards agenda has been given expression in many parts of the world- for example the development of quality frameworks and the emergence or refocusing of regulatory bodies –distinctly individual national priorities, activities and support structures have also emerged.

An investigation of standards-related projects supported by the (recently abolished) Austra-lian Learning and Teaching Council (ALTC) was undertaken as the situational analysis stage of the Assessing and Assuring Graduate Learning Outcomes (AAGLO) project (Barrie, Hughes and Crisp 2011). Though the issue of ‘standards’ had been a central or peripheral concern of many ALTC pro-jects since its 2004 inception (Oliver 2011), the Learning and Teaching Academic Standards (LTAS) project (ALTC 2011a) was the first ‘standards’ project coordinated on a national scale.

This paper briefly introduces the LTAS project and Australia’s evolving higher education regu-latory arrangements (TEQSA 2011) in order to contextualise an analysis of the large and diverse range of activity the ALTC has subsequently encouraged. The analysis of 11 ALTC grant projects and four fellowships that are currently addressing academic standards focuses on factors such as discipline, scope, sponsoring or leading body, the nature of outcomes and approaches to the conduct of the project and to the assurance of standards. The analysis makes sense of the multiple approaches and perspectives being delivered through this range of activity and adds a comple-mentary perspective to the standards discussions in the Australian higher education context.

Paper 2.- Current Australian approaches to assessing and assuring academic standards.

Paper 3.- Learning to mark: exemplars, dialogue and participation.


33

PaperDevelopment of an Assessment Technological Literacy.

Jan Ardies • Bruno Bossaerts • Sven De MaeyerAn increasing importance of broadly educated youngsters is emerging in Europe. Several coun-

tries (e.g. UK, The Netherlands, Belgium,...) are redesigning their national or regional compulsory curricula. Thinking about the need of science and technology in the standard curricula is a dis-cussion started in the seventies or even before in most countries (Fensham, 2004). Although the growing interest in the field of technology education is relatively young and little research has been done (Garmire & Pearson, 2006). Technology as a school subject is now in most countries mandatory in secondary and/or primary education. Nevertheless there is little research done about educational effectiveness in technology classes and the state of technological literacy of pupils. This research is aiming to develop a test to examine the dimensions ‘understanding’ and ‘making sense’ (Van Woensel, 2010) of pupils in order to make it possible to measure student outcomes concerning technological literacy. Several phases are undertaken to develop the test. Focus groups and individual interviews with experts in the field of technology education deter-mined a starting point and provided a set of questions. These questions were analyzed, mainly through the Item Response Theory based on a pilot study (n=251). A second round of analyzing the data based on new content analysis and a new sample of 2000 students resulted in a final test providing information about the status of technological literacy of this representative group. In this study we will report on analyses on the fit of the test-data with suitable IRT models and on the selection of proper items from a pilot test using a combination both the content-based and IRT-perspective. Finally we will report on a DIF-analysis, answering the question whether certain items function different for boys and girls.


Paper Integration of e-assessment processes for large-scale institutional development.

Carmen Tomas • Michaela Borg • Jane McNeilTen years ago, e-assessment was predicted to become all-pervasive in higher education insti-

tutions. A combination of cultural, political, resource, and infrastructural constraints has resulted in a much slower take-up than expected. The current investigation aims to address the issue of

on scholars such as Sadler, we examine the suggestion that exemplars are important for newco-mers coming to understand local assessment practice. We argue for the fundamental importance of dialogue about exemplars and other aspects of practice – both to develop inter-subject un-derstandings of assessment ‘anchor points’, and also to foster among newcomers (and especially contract lecturers) a great sense of participating in, and being valued by, the local assessment community.

Shay, S. (2004). The assessment of complex performance: a socially situated interpretive act. Harvard Educational Review, 74(3), 307-329.



34

Paper Effects of strong portfolio triggers of heuristic instruments and linked summative/formative assessment on writing, writers and tutors.

Kris BuyseIn this paper we describe two instruments in the assessment of writing, i.e. (1) strong portfo-

lio triggers for using heuristic instruments such as spelling checkers during the writing process and (2) a strongly linked summative/formative assessment, assessing the (summative) exams in exactly the same way as the (formative) take-home tasks. After outlining the context of writing as-sessment that gave rise to the experiments with the two instruments, we will present a first range of possible effects on the writing process and products of higher education students in Flanders, as well as on the writers and the tutors, i.e. how these assessment procedures affect and enhance student learning, writing assessment and organizational change.

The results of a survey as well as those of the analysis of the scores’ evolution suggest that the implementation of the first instrument is motivating and effective for writing activities and cour-ses among the students: it not only improves their writing products, but also enhances better and motivating writing and learning processes in general. Furthermore, it also affects the writing as-sessment of tutors, and indirectly organizational change, as they asked (and obtained) a different implementation of the writing course (in terms of conditions with regard to frequency, audito-rium and evalution system), based on the positive results of their methodology .

With regard to the second instrument, the same quantitative and qualitative analysis will be

e-assessment take-up from an assessment life-cycle perspective, with view to establishing a set of priorities for institutional development.

Twenty-six academic members of staff were interviewed on their uses of technology to support assessment, in order to develop an institutional perspective and to identify the barriers that pre-vented more larger scale use of e-assessment technologies.

The interviews revealed extensive and invisible uses of technology in development of the task (using email, internet etc) and communication stages (via VLE) of the process. The use of technolo-gy to communicate assessment to students was largely the result of a previous institutional policy.

The assessment event was found to vary according to different applications. Use of e-portfolios was in the main limited to certain disciplines and reflected limitations similar to its paper-based counterpart. Another e-assessment application, computer based assessment (CBA) encountered both infrastructural limitations and a perception that its suitability is at lower levels of study. These combined constraints relegated CBA to formative and low-stakes assessment.

Electronic submission of coursework was reported to be subsidiary to paper processes mainly for cultural reasons and practices. Whilst the technology means were available, cultural barriers around marking prevented the establishment of full electronic submission, marking and fee-dback. The other large scale application, exams, was limited by the infrastructure.

Given these barriers, stages in marking, entering feedback, and storage of assessment records and archiving were primarily paper based. The assessment life-cycle perspective on e-assessment facilitated insights into which areas are most problematic and which applications could most ra-pidly bring more gains to an institution.



35

PaperRubrics as a link between standards and formative assessment.

Robbert SmitThe aim of the presented study is to examine the learning benefit of rubrics as part of the pro-

cess of formative assessment. We see rubrics as a beneficial tool to align lessons with standards (Burke, 2006). The rubric used in this project is embedded within the topic of “reasoning in mathe-matics” and in line with the Swiss standards of mathematics (Linneweber-Lammerskitten & Wälti, 2005).

Items in large-scale tests don’t allow for just a few open-ended questions with extensive written answers, as they need a sufficient number of items in order to reach satisfying reliability values. Nevertheless important mathematical competencies like problem solving or reasoning require time to search and explain. So it lies with the teacher to test these competencies in more authentic

Paper Friendship in the reliability of non-rubric and rubric-based peerassessment.

Ernesto Panadero • Margarida Romero • Jan-Willem StrijbosThere is an increase in the use of peer assessment in collaborative learning activities. One of the

challenges for the peer assessment acceptance among learners and teachers is the reliability of this procedure. In this study we explore the use of a rubric to enhance the reliability of the peer assessment. The rationale is based on the criteria and scoring features included on the rubric that could contribute to better assess a peer work. The social aspects of peer assessment are also stu-died by analysing the effects of the friendship between the evaluator and the assessed peer. The study involved 216 pre-service teachers from the Universitat Autònoma de Barcelona in their third year. A quasi experimental design was used where half of the students evaluated a peer concept map using a rubric while the other half evaluated without rubric. The results show a generalized tendency to over-score the peer works. Nevertheless, an increase of reliability in the rubric group was found when compared to the students’ assessing without the rubric, with the latter scoring significantly higher their peers’ work using the score given by the expert/teacher as the reference. To analyze the impact of friendship on the accuracy of the score students informed about their friendship degree towards the student they were assessing. Three friendship degrees were consi-dered (low, medium and high). For low and medium friendship degrees the students using a ru-bric-based assessment made a more accurate assessment than the non-rubric group. For the high level of friendship no differences were observed between the rubric and non-rubric students. We discuss the interest of the rubric-based assessment in the context of peer assessment with a spe-cial interest on the effects of this methodology for students showing low and medium degrees of friendship.

performed on the data of the current academic year 2011-2012, so that we are not able yet to an-swer the question, but at first glance the data of the first semester look promising.




36

PaperScoring rubrics for student learning.

Anders Jonsson • Ernesto PanaderoScoring rubrics seem to have the potential of promoting student learning and the gains in stu-

dent performance when using rubrics are sometimes conspicuously large. Still, besides the fact that there is a quite limited number of studies on the formative use of rubrics, there are also seve-ral limitations to this research. Starting from these limitations, this paper reviews current research on formative uses of rubrics. By reviewing such research both in greater detail and with a primary attention to influences on student learning, the aim of this paper is to investigate in what ways rubrics may support student learning and also to identify factors that may influence – either po-sitively or negatively – the effects of using rubrics to support student learning. Findings suggest that the use of rubrics may support student learning by: (1) aiding teachers in improving instruc-tion, (2) aiding the students in determining teacher’s expectations, (3) aiding the students in de-veloping self-regulated learning, (4) facilitating the feedback process, (5) lowering student anxiety about assignments, and (6) improving students’ self-efficacy. Findings also suggest that while the studies performed in higher education contexts report on positive results when providing the students with rubrics, the positive results in schools are linked to longer and larger interventions. One factor that seems to be of paramount importance, for school students, is therefore the time devoted to work with the rubric. There is also a apparent difference between studies reporting on small or partial effects of using rubrics on the one hand, which to a large extent use rubrics only for self-assessment, and studies reporting on larger gains, where a broader repertoire of instructional interventions are used. When it comes to rubric design, not many factors have been systematically investigated. However, findings suggest that rubric design may not be crucial for student learning.

and appropriate situations. Standard based rubrics can help to make criteria known to students and to assess mathematical reasoning.

Since our research project had the character of a pilot study we limited ourselves to a small sample of four primary classes. On the one hand we used quantitative methods to measure the students competency in mathematical reasoning. On the other hand we interviewed the teachers and groups of students to gain information about the success of our lessons with rubrics as a sup-porting learning tool.

Using Rasch-modeling we were able to validate the four levels of our rubric, but the reliability of the students’ competency level measures is rather low due to the few open items. The qualitative content-analysis of the interviews shows that all students have generated a basic understanding of the competency. They gained the insight that reasoning is not just wrong or right, but it can be done on different competency levels. The students have learned to self-assess themselves and the teachers felt supported by the rubric when giving feedback.



37

Paper A Feedback Intervention Theory Perspective on External School per-formance Data Use in German Secondary Classrooms.

Carolin Ramsteck • Annette FruehwachtDespite growing research literature on the effects of mandatory testing and performance fee-

dback systems, there is still a lack of theoretical understanding on how performance feedback works to improve instruction and student learning. This paper aims at exploring feedback inter-vention theory (FIT) as conceptual framework for modeling instrumental feedback data use in German secondary classrooms. Four FIT-related assumptions on effective external performance feedback data use were applied to qualitative teacher interview data. Data were gathered in 20 German grammar schools. The material was analyzed with qualitative content analysis methods (deductive category application in combination with inductive subcategory building). Results show that only some teachers are able to assign performance feedback data to specific aspects of their teaching (as focal task). Furthermore, feedback data does not operate as corrective feedback since teachers are not able to test fine-grained didactical hypotheses with feedback reports at hand. The paper concludes that FIT might be a valuable perspective to further investigate the flaws and potentials of current mandatory testing policies in Germany and other European coun-tries.

Paper Providing formative feedback to students via emails and feedback strategies related to student metacognition.

Chang ZhuAssessment is a central element in the overall quality of teaching and learning in higher edu-

cation. Formative assessment aims to enhance learning through providing appropriate feedback. Effective formative assessment helps the student to see the gap between their current understan-ding of course components and the goals of the course so that they can take the appropriate ac-tion to achieve these goals. Formative feedback can be provided to students through a variety of ways. Among all those methods, electronic feedback methods are increasingly used by teachers since they enhance feedback production, delivery and communication (Race, 2001). Electronic feedback has the advantage of providing timely and individualised feedback. Among the factors related to student learning, metacognition is a very important one as it makes the learner aware of his/her cognition and trigger control processes that serve the pursued goal of the self-regulation process (Efklides, 2008).

This research focuses on studying students’ preferences of formative feedback strategies and the effects of teacher’s formative feedback on student learning. Participants in this study were third-year undergraduate students (n=109) studying Business Management in a university in Bei-jing, China. The findings of this study indicate that providing the right kind of feedback to stu-dents can make a significant difference in their achievement. In addition, student characteristics such as metacognition level need to be considered in the adoption of different feedback strate-

11h-12h30: Breakout room 1Dollar



38

Paper Computer based assessment of driving competence involving different response modes.

Sarah Malone • Sarah Katharina Freidinger • Roland BrünkenOne empirical indicator of test validity is a clear difference between expert and novice test per-

formance. Some general findings of expertise research may help to develop such valid tasks. One of these findings is that experts outperform novices more clearly with increasing domain specifi-city of the given task–it’s similarity to the characteristic demands of a defined domain–(Glaser & Chi, 1988). This aspect implies that in order to make a statement on a participant’s skill in a cer-tain domain, the individual should be given a task that resembles the tasks that naturally arise in that domain. Some domains like e.g. sports and driving require fast decision making under time constraints (Hodges, Huys, & Starkes, 2007). For the assessment of expertise in such domains test items including an adequate response format seem to be crucial for the ecological validity of the test. The aim of the present study was to investigate whether the validity of the assessment of expert performance in car driving can be enhanced by an ecological valid response mode of the applied tasks. 35 learner drivers (novices) and 31 individuals with some driving experience (ex-perts) were to solve 21 items including identical animated driving scenarios. The response format for each scenario was varied experimentally: one half of the participants were to react to breaking hints within the animation (high ecological validity) and the other half worked on multiple-choice questions following the presentation (low ecological validity). We postulated that driving experts would outperform driving novices in both response modes. Moreover, the hypothesis that follows from the concept of domain specificity is that the mentioned expert-novice difference should be more obvious for the reaction time response format. Whereas the ANOVA revealed a main effect of expertise, no interaction between the factors expertise and response mode could be found.

gies. Feedback via email communications was considered a very convenient and timely manner for the teacher and students in this study. The current study finds that students with high and low metacognition reported a different level of preferences for feedback strategies; nevertheless most students prefer constructive and elaborative comments from teachers. Furthermore, the results show that teacher’s formative feedback was appreciated by students, and promoted student lear-ning.

Paper Complex Problem Solving in Educational Contexts –Assessment, Measurement Invariance, and Construct Validity.

S. Wüstenberg • S. Greiff • G. Molnár • A. Fischer • J.Funke • B. CsapóComplex Problem Solving (CPS) is seen as a cross-curricular competency, which has recently at-

tracted interest in large-scale-assessments. In the PISA 2012 cycle, CPS will be assessed across the world partly using minimally complex computer simulations based on the MicroDYN approach.




39

In this talk, we present empirical results on MicroDYN, a computer-based test containing multiple items aimed to measure two main complex problem solving processes - knowledge acquisition and knowledge application. More specifically, we (a) test a measurement model for MicroDYN composed of the two aforementioned processes, we (b) evaluate whether measurement invarian-ce holds across different grade levels of high school students in order to compare latent means between grade levels and (c) investigate relations between CPS, reasoning, grade point average (GPA), and parental education. Analyses are based on N=855 Hungarian high school students in grades 5 to 11. Using structural equation models, results show that (a) a 2-dimensional model with the facets knowledge acquisition and knowledge application fit the data best. Furthermore, (b) CPS is strongly factorial invariant and thus, mean differences between grades can be interpreted meaningfully. Latent means in both facets increase at higher grades, disregarding a considera-ble drop in CPS performance in grade 9. Finally, (c) results based on path analyses indicate that knowledge acquisition incrementally predicts variance in parental education and – to a lesser extent – in GPA even beyond reasoning. A proper assessment of CPS in addition to subject related abilities is of high educational relevance. Results of this study provide important implications on how to use CPS in an assessment context.

Paper Multiple perspectives on a viva voce as final assessment in veterinary science.

Gordon Joughin • Michael McGowanThe viva has a long history. It has been, and remains a prominent form of assessment in many

European countries and is used across a wide range of disciplines in the UK, Australia and el-sewhere. The final year veterinary science program at an Australian university utilises a viva as the final piece of assessment in a five-year program. Students are assessed on their ability to com-petently investigate clinical case scenarios and communicate practical, economically responsible and scientifically valid recommendations on diagnosis, clinical management and control of a di-sease problem. Multiple perspectives on the viva were sought through the following process: (i) A random sample of students were interviewed immediately following their viva, with questions focussing on their expectations of what would happen, what they believed would be tested, how they prepared for the viva, how they would compare the viva with written examinations, and whether they would recommend it as a form of assessment. (ii) The four academic staff who con-ducted the viva in pairs were also interviewed about their perceptions of the viva, its purpose, advantages and disadvantages. (iii) The Faculty’s Associate Dean (Academic) was interviewed re-garding his perceptions of the viva. The three sets of perspectives of student, academic, academic administrator are presented in this study. Each set of perspectives serves to illuminate our unders-tanding of the viva as an engaging and revealing form of assessment. Comparing and contrasting the perspectives reinforced the value of the viva as a ‘rite of passage’ from the perspectives of all stakeholders, with students also drawing attention to the impact of the viva on their approaches to studying.



40

Paper Quality improvement of summative radiological skills assessment with digital multi-slice images in medical education.

Cécille Ravesloot • Marieke van der Schaaf • Olle Ten Cate • Jan van Schaik • Anouk van der Gijp • Koen Vincken

Background:Current radiology practice is based on the digital interpretation of volumetric multi-slice (MS)

images (like CT and MR images), however assessment of radiological skills in undergraduate me-dical education is still based on two-dimensional (2D) images. Consequently, the assessment lacks authenticity, which negatively impacts validity. We hypothesized that the quality of assessing ra-diological skills can be increased using MS images.

Methods:In 2010 322 medical students, trained with MS images took a digital radiology test (Version A

and B), containing eleven 2D and ten MS image items (CT-scans). Participants filled out a ques-tionnaire concerning authenticity of the assessment as an indication for validity. Mean scores and reliabilities (Cronbach’s alpha) of the 2D and MS image subtests were compared.

Results:Cronbach’s alphas (corrected with Spearman-Brown formule) on 2D image items (k = 40) were

.52 (Version A), and .68 (Version B) and of MS image items (k = 40) were .79 (Version A), and .78 (Version B). This difference in reliability estimate between MS and 2D image items was significant for Version A.

Scores on MS image items (M 8.0 SD = 1.5; M 7.6 SD 1.5) were significantly lower than scores on 2D image items (M 9.7 SD = 1.1; M 9.0 SD 1.3) (t (df138; 142) = 1.7; 1.5, p < .001; r = .24; .18). Assessment based on multi-slice images was considered more authentic (t (271) = -3.81, p < .001).

Conclusion:According to the participants, assessment with MS images increases authenticity, which can

contribute to validity. Reliabilities were higher for items with MS images compared to items with 2D images. This might indicate that MS image items are more difficult than 2D image items, and therefore more discriminating. Further research with larger tests and studies on participants’ cog-nitive processes involved are necessary.

Keynote Main conference room - Olympia Bourse BallroomUp or Out and the Joy of Lifelong Learning.

Menucha Birenbaum

The presentation offers a conceptualization of assessment culture (AC) in school that stems from complexity thinking. It highlights two complex systems that are nested in the AC system, na-mely classroom learning and teacher professional learning, and points to fractal relations among the three systems and to the recursive interactions whereby they influence each other in a co-evolutionary way. It then depicts the nature of AC and the mindset it reflects, as contrasted to that of a testing culture. Consequences with regard to school locus of control, regulation, and adapta-




41

tion to external demands (accountability) are addressed, and recommendations for fostering and researching an AC are offered.

SymposiumReconceptualising Feedback in Higher Education.

This symposium uses the lens of assessment feedback to examine how to better support stu-dents’ learning. Feedback is an essential part of learning but is often regarded as an end product rather than part of an on-going cycle of learning. Its generation is also largely assumed to be the responsibility of tutors rather than the learner or their peers. By taking a different perspective on feedback we challenge these assumptions, beliefs and resulting practices. Drawing on Feedback: an Agenda for Change, drawn up by a group of international experts (researchers and practitio-ners) at an ASKe event in 2009 the symposium questions the current notion of feedback as pro-duct. It proposes the reconceptualisation and repositioning of feedback within a more coherent paradigm of student learning and assessment which recognises the processual, relational and dia-logic dimensions of effective feedback (e.g. Nicol, 2009) as well as its role in developing students’ evaluative abilities (Boud, 2006).

Largely using theoretical arguments the papers address the need for change and ways in which it can be realised. The symposium will be framed by a paper setting out the ideas and rationale of the Agenda and its calls for change. This is complemented by a paper that explores the nature of feedback through its multiple definitions and the role of self assessment in relation to satisfying the meanings of those definitions in practical terms. The third paper looks at feedback and the learning community, examining the perspectives of staff and students and goes on to develop a process model of self assessment.

In particular these papers illustrate the close relationship between feedback and learning and make arguments to reposition feedback centrally in the process of learning rather than an isolated and final product.

Symposium Chair: Margaret Price Symposium Discussant: David Carless

Margaret Price • Karen Handley • Berry O’Donovan • Chris Rust Jill MillarEffective feedback is a crucial element of assessment and has great potential to enhance lear-

ning. Yet many current practices across Higher Education do not fulfill this potential. Shortcomings in feedback are evident in student dissatisfaction with existing approaches. This paper addresses the growing impetus for change and proposes that to be effective change must be fundamental, involving a reconceptualisation of feedback.

Responding to this impetus a group of international researchers and experts in the field (known as the Osney Grange group, OGG) came together to critically examine and discuss the theory and practice of feedback. They concluded that current practices in Higher Education are often not fit for purpose being largely founded on myths, misconceptions and mistaken assumptions that undermine student learning. For example, one persistent myth is that tutors are the only ones


Paper 1.- Assessment Feedback: Agenda for Change.


42

who can give feedback and this largely precludes the use of self and peer assessment that would enable student to develop evaluation skills, a key graduate attribute. Furthermore, current prac-tice is over-reliant on written, unidirectional, tutor-delivered feedback which discourages discus-sion about the meaning and application of feedback. There is a need for stakeholders to re-think the role and purpose of feedback and consequently the OGG developed Feedback: an Agenda for Change (available at http://www.brookes.ac.uk/aske/OsneyGrangeGroup).

The Agenda seeks to promote a re-conceptualisation of feedback from product to process and, in doing so, recognises the importance of its relational and dialogic nature. Feedback needs to have a long-term relational dimension and be supported by dialogue with tutors and peers hel-ping students to better interpret feedback, which is often ambiguous. The Agenda identifies the importance of students ability to use varied sources of feedback and evaluate their own work and that of others. This paper presents, explains and justifies the clauses of Agenda for Change.

Maddalena TarasDifferent educational contexts have resulted in diverse assessment developments in both

theory and practice. Higher education is self-contained and accredits, assesses and awards its own qualifications giving it overall control of formative and summative assessments. The compulsory sector on the other hand, has mainly externally accredited, assessed and awarded qualification with some internally assessed and moderated course-work and therefore sees most of its summa-tive assessments for accredited work out of its immediate control.

Therefore, over time, different communities of practice have developed strategies to support and understand their own contexts and these are manifest in how discourses, definitions and processes are understood and presented.

A superficial commonality of terminology is misleading and potentially dangerous because it is easy to assume that we are speaking the same language when in reality we are referring to di-ffering conceptualisations and epistemologies. This paper examines three aspects of assessment across educational sectors: firstly, definitions of feedback, secondly the relationship between fee-dback and formative assessment and thirdly, how feedback is construed in the five self-assessment models identified by Taras (2010). Feedback is an aspect of assessment which reflects clearly the link between learning, teaching and assessment. How it is construed highlights the roles of the protagonists and how they interact and relate in order to support institutional learning.

The internationalisation of communications through publications, conferences and personal contacts has dissolved barriers across educational sectors requiring an understanding of discour-ses and practices. Transfer of ideas and practices are inevitable and therefore a clear understanding of how theories and practices are context specific is necessary in order to avoid misunderstanding and misuses of work across sectors. By understanding the aspects which are context-specific it is possible to transfer innovative thinking and practices, adapt them to new contexts and maximise efficiency of transfer (Taras 2008).

Paper 2.- Feedback on Feedback: uncrossing wires across sectors.


43

Stephen Merry • Paul Orsmond • Karen Handley

Paper Academic assessment centers and student competencies: Reliability, construct, consequential and predictive validity.

Rutger KappeThe paper reports on the use of assessment centers (AC) in higher education. Students (N = 137)

in the human resource management program participated in two AC’s; once at the end of their first year and then again at the end of their third year of study. They were assessed on nine compe-tencies. In addition overall GPA as well as five specific GPA measures; lectures, skills training, pro-jects, internships, thesis were available. Years later, after graduating from college, students were contacted and asked to provide self-ratings of career success.

Results show that, a) competence were measured with high levels of inter-rater reliability, b) the nine competencies could be collapsed into three competency dimensions and an overall factor,

The purpose of this presentation is to explore, within the context of social learning, the conse-quences of feedback practices in Higher Education and to suggest ways feedback can be imple-mented in terms of enhancing student learning outside the formal curriculum.

It has been shown that students develop communities of practice that function in accordan-ce within Wenger’s conceptual framework and particular patterns of participation are displayed. Consequently, an awareness of community practice would seem to be a necessary condition for understanding the student learning that results from tutor feedback. Tutor feedback needs to be reconceptualised, being seen not as communication between an individual tutor and an in-dividual student, but as communication between an academic community and a student com-munity. Translation of language occurs with student communities, perhaps, reinterpreting the intended messages of tutors. Students also engage in peer-assessment through their ongoing practice with others who participate differently in the community, and distributed cognition and self-assessment practices emerge from the peer feedback generated in this way. Each community member negotiates meaning in their individual ways.

Understanding student community practice is also important because it may provide opportu-nities for tutors to enrich the student learning that naturally occurs outside the overt curriculum. If feedback is seen in its broadest sense as encompassing peer and tutor discussions, focussed dialogic teaching practices provide opportunities for situated learning within the curriculum such that common understandings are generated which serve to minimise differences between aca-demic and student communities. Research informed teaching, exemplars and the GOALS process may be suitable vehicles such classroom dialogue. Tutors are also, thorough suitably designed feedback practice, able to enrich and develop students’ self-assessment qualities thus enhancing their learning abilities.

Paper 3.- Students’ social learning and its implications for feedback.



44

c) students received higher AC ratings in their third year, d) the correlation between GPA and AC ratings was significantly higher for third, compared to first, year students, e) AC ratings showed the highest correlation with ratings for internships,

and f ) students perceived the AC experience as being helpful to their learning. Concerning the predictive validity of the academic AC the findings show that competency ratings were the best predictor of early career success, even after holding GPA constant, however the reverse was not true. Whereas AC’s measure broader competencies that are learned throughout the curriculum, traditional exams measure short term, task or course related performance. Because AC’s can be cost effective, educational institutions should consider using existing exams and AC’s alongside one another other.

PaperSchool quality evaluation: students’ points of view.

Valentina Grion • Séverine ColinetAssessing school quality at a national level is a current policy fashion (Broadfoot, 2009) in most

European countries and, at the same time, is a subject widely discussed by researchers. Today, many assume a critical position toward the evaluation systems linked to the Measurement Pa-radigm (Broadfoot, 2009) and highlight its limits and pernicious effects (The Cambridge Primary Review, 2009; Stobart, 2008). They claim to employ strategies to remedy deficiencies of the exces-sively pervading use of performance testing. This might be interpreted as a need to use a greater diversity of tools and multiple sources of data, in order to obtain an effective system of evaluation in schools.

Some authors affirm the importance of measuring client satisfaction and perceptions (Elliot&Sammons, 2000); Walford (2002) asserts that there are two types of client whose voices have

not been listened to sufficiently, the first includes young people who experience school on a day-to-day basis.

In this framework, we are conducting an empirical work to verify the possibility of establishing a set of evaluative indicators, based on students’ representations of “a good school”.

In the first phase, a purposive sample of about 150 students was chosen from a French andan Italian middle school. During normal class time, participants answered a written open ques-

tion which asked them to define “a good school”. Using a bottom-up approach, we conducted a content

analysis of the texts, in order to identify emergent categories of content and to generate meanings from the students’ answers.

The results should highlight the similarities and specificities of each country in order to findcommon indicators for evaluating the quality of schools in a transnational perspective.Although our analyses are at a very early stage, we think there are some interesting aspects to

be discussed.



45

Paper Do different assessment routes (linear vs. modular) prepare students, in the same way, for further study?

Carmen Vidal RodeiroThe modularisation of GCSEs (qualifications taken by students aged 14-16 in England) has cau-

sed considerable controversy since its introduction. There are a number of different perspecti-ves relating to the modular vs. linear assessment debate. Firstly, there are those who believe that modular assessment could lead to lack of coherence and fragmentation of learning as students have little time for reflection, skill development and knowledge consolidation. Secondly, modular assessment might not provide opportunities for deep learning and might, instead, encourage a climate of cramming. Thirdly, the increased assessment load might encourage students to spend more time revising for the modular exams, rather than simply benefiting from the learning expe-rience. Fourthly, there is the view that re-sitting modules may be lowering examination standards, including ‘teaching to the test’ time heightened at the expense of deeper learning.

Based on the above issues, teachers expressed concerns about modular students being less well equipped for the transition from GCSE to further study (e.g. A-levels, qualifications taken by 18 year-olds) than their linear counterparts. This study set out to investigate whether modular cour-ses are good preparation for further study. The focus was on the impact of the GCSE assessment route on the uptake and performance in three A-levels: English, mathematics and ICT.

The research showed that students following a linear assessment route at GCSE were more likely to continue to study mathematics at A-level than those who followed a modular route. Con-versely, linear students were less likely to progress to A-level in ICT. There were no differences between linear and modular students in the uptake of English at A-level.

Once attainment at GCSE was taken into account, the assessment route only affected perfor-mance in mathematics, with students following a linear route at GCSE performing better at A-level than those following a modular one.

PaperEarly literacy assessment and Intervention.

Julie Brosnan • Elisabeth Scherz • Jesse Butler • Olive Healy • Kendra Brooks-Rickards

A Response to Intervention Framework is an all-inclusive early detection and prevention stra-tegy designed to identify and support struggling readers at the first sign of difficulty (Gersten et al. 2009). RTI frameworks operate through a multi-tiered approach; this study investigates the effectiveness of a Tier 2 intervention designed to support the development of Senior Infant (kin-dergarten) student’s early reading skills. The Tier 2 Intervention combines Children’s Progress Aca-demic Assessment (CPAA) as the universal screener to identify the pool of Tier 2 students, with an intervention defined as precision teaching targeting the component skills related to reading (e.g. letter sounds, letter identification, initial sounds, final sounds etc.). Thus the study focuses on an




46

Paper Effects of Self-Paced Learning on Achievement in Sentence Level Work in Year 5.

Mary Sheard • Bette ChambersA development in the use of learner response devices (LRDs)allows pupils to answer questions

at their own pace, providing instantaneous feedback to the pupils and their teachers about the pupils’ understanding of the concepts just taught. Teachers should then be able to provide diffe-rentiated support to pupils, improving their learning. A small-scale randomised evaluation of the self-paced learning strategy (SPL) was conducted to determine if this form of technology–sup-ported formative feedback increases pupil learning. Forty-two Year 5 classes in primary schools in the North of England were randomly assigned to experimental (21 classes) or control (21 classes) conditions for a 12-week trial period during the Spring term, 2012. An additional 16 Year 5 classes who were already experienced in using LRDs participated as a matched comparison user group, so providing a second experimental group. Learning Clip, an interactive resource developer, pro-vided 40 question sets with a grammar focus for the experimental groups to implement in their literacy lessons during the research period.

Equivalent pre- and post-tests reflecting the Year 5 sentence-level content were the measures used. Training and support in SPL using ActiveExpression learner response devices was delivered by Learning Clip personnel to teachers in the experimental groups in their schools and reinforced by ongoing support.

Results from a similar previous study in mathematics showed no statistically significant diffe-rence between pupils’ scores at pre-test and a statistically significant difference in their post-test scores. In reporting findings from this more recent study, the present paper draws parallels with the earler findings and makes recommendations for maximising the potential for technology-supported self-paced learning in increasing primary children’s grammatical achievement. In addi-tion, the paper suggests how the empirical analyses relates to an extended theoretical model of formative asassessment in the context of educational effectiveness.

assessment-intervention-evaluation cycle within the applied setting.The study will run for 8 weeks, with approximately 15 ‘senior infant’ (kindergarten) students

ranging in age 5-6.10 years serving as participants. The setting is a primary school located in a rural area in the west of Ireland. All 15 senior infant students will be administered 4 different types of assessments related to early literacy skills (Children’s Progress Academic Assessment; Dynamic Indicators of Basic Early Literacy Skills; Curriculum Based Measures; The York Assessment for Rea-ding Comprehension). Those Senior Infant students who score ≤2 on Children’s Progress Acade-mic Assessment subtests of Phonics and Writing, and Phonemic Awareness will be selected for Tier 2 intervention for the duration of the study. Each student will receive 15-20 minutes daily of targeted precision teaching procedures during the school week. Tier 2 intervention effectiveness will be investigated within a multiple baseline experimental design. Data will be analysed through visual inspection and analysis of the data; estimation of effect sizes; and calculation of ratio gains for outcome measures. Results will be available in March 2012.



47

PaperPractical problems in the assessment of text comprehension.

David Hidalgo • Jesús Alonso-TapiaAssessment influences learning activities. So, it is necessary to investigate whether teachers as-

sess knowledge and competence adequately. In this study the objective was to identify strengths and shortcomings in teachers’ ways of assessing text comprehension. With this purpose, after es-tablishing the psychological models from which ideas on text comprehension assessment should be evaluated, a sample of 111 Spanish Primary School teachers were asked to design assessment tasks in relation to two texts of different structure. The tasks should be adequate for deciding the kind of help to give the students to enhance their text comprehension. The lack of regular as-sessment patterns emerging from results showed that teachers’ practice does not correspond to any shared model. Moreover, results also showed specific strengths and shortcomings of teachers’ ideas on text comprehension assessment, as well as the degree in which the kind of text influen-ces assessment practices. Implications for designing teachers’ training on students’ assessment are discussed.


Paper The impact of a dedicated assessment course on developing teacher education students’ assessment capability.

Mary Hill • Alison Gilmore • Lisa Smith • Bronwen CowieThe primary aim of the research was to explore how primary teacher education students in three-

year programmes across four universities in New Zealand develop their assessment knowledge, beliefs and practices (assessment capability) during their teacher education programmes. In all four universities the assessment curriculum is embedded within curriculum and professional practice courses as well as practicum experiences in schools. However, in the second year of stu-dy, two universities provide a dedicated assessment course, as well. Participants were the 2010 entering cohorts (n = 711) at the four universities. The second-year students in two of the univer-sities with the assessment course (n = 289) were compared with those in the universities that did not have a dedicated assessment course (n=282). Data were collected through a Beliefs About Assessment questionnaire administered at the beginning of year one and at the end of year two. The questionnaire contained six open-ended questions and 46 Likert-type questions based on a conceptual framework of assessment capability developed from a position as naïve student to emerging professional.

Analyses of the first year students’ assessment knowledge, beliefs, and practices established that their views about assessment were ‘naïve and largely represented their own experiences of being assessed as students themselves. There were no differences between universities in ente-ring students’ views. However, analyses of students’ views after their second year of study revea-led emerging understandings of assessment consistent with being a teacher. In responses to the Likert items, there appeared to be little impact of students taking a dedicated assessment course. However, qualitative analyses of the open-ended questions revealed differences in the views held by the two groups of students.



48

Paper Student and Teacher Educator Perceptions of Becoming Assessment-Capable Teachers.

Lisa Smith • Bronwen Cowie • Alison Gilmore • Mary HillThe primary aim of the research was to explore what primary teacher education students in a

three-year programme across four universities in New Zealand believe and understand about as-sessment at entry, during, and at exit from their programmes. Participants were the 2010 entering cohorts (n = 711) at four universities. Third-year exiting students (n = 289) at the four universities provided a comparison group. In addition, at each university, n = 6-10 entering student partici-pants took part in focus groups and n = 8-12 teacher educators took part in professional conver-sations. Data were collected through a “Beliefs About Assessment” questionnaire, semi-structured protocols for the student focus groups, and semi-structured protocols for the teacher educator conversations. Analyses of the questionnaire items comparing the entering cohort to the exiting cohort indicated that significant changes to beliefs about assessment occurred over the course of 3 years. There was progress from a naïve view that assessment is mainly formal and summative in nature, toward an emerging professional view that highlighted the need to understand the role of children in assessment for learning. In the teacher educator conversations, five themes emerged: an emphasis on teaching about formative assessment, noting the need to shift students’ thinking from summative to “next steps,” the need for students to understand using multiple sources of evidence, learning how to make judgments, and having strong curriculum knowledge. The stu-dent focus groups showed beginning understandings of formative assessment and brought out

Paper The electronic portfolio as an instrument for the development of teacher competences in pre-service teaching practice.

Katrien Struyven • Yves Blieck • Veronique De RoeckDuring recent years teaching practice has gained importance as a significant period of learning

for the development of teacher competences. These teacher competences are regarded as the possession and development of sufficient skills, knowledge, appropriate attitudes and experience for the successful performance of one’s professional roles as a teacher (McNamara, 1992). Hen-ce, many teacher education institutions are aiming at the development of teacher competences (also) during teaching practice. A popular instrument to achieve this aim and to assess the develo-pment of teacher competences is the electronic portfolio. This research study investigates the use and effects of a competence-based electronic portfolio for the assessment of teacher competen-ces during teaching practice in a Flemish academic teacher education program. Therefore, ques-tionnaires and semi-structured interviews were administered to three parties, student teachers (N=41), teacher trainers (N=25) and mentors (N=46). Results show that the portfolio proofs to be a useful instrument for the assessment of teacher competences according to teacher trainers and mentors. Student teachers, in contrast, do not belief that the portfolio achieves this aim. In their views, the portfolio is unable to reflect and demonstrate their teacher competences sufficiently. Why that is the case, is the central focus of this paper.




49

the impact of practicum experiences on student thinking and practice. The discussion focuses on what teacher educators need to be aware of in providing for students in the early stages of their education.

Paper A Validity Framework for Assessments Designed to SupportInstruction.

James Pellegrino • Louis DiBello • Susan GoldmanThe overarching goal of the work we propose to report involves implementation of a compre-

hensive, multidisciplinary approach to the design and validation of assessments that are intended to function close to instruction, i.e., assessments designed for both summative and formative pur-poses. The goal of the work is to transform how such assessments can be used in STEM classrooms and enhance their instructional effectiveness and impact on student learning. Our multidiscipli-nary approach integrates disciplinary knowledge of STEM content, experience in STEM area tea-ching and learning, cognitive science, educational measurement, and psychometric modeling. The cases we are working on include assessments embedded in K-8 mathematics curricula, tech-nology-based diagnostic assessments for 6-12 physical science, and concept inventories used in undergraduate engineering courses.

The project’s research questions and associated data collection flow from a framework we have developed for studying the validity of assessments that are intended for use in instructional con-texts. The validity framework describes three major components labeled as: Cognitive, Instructio-nal, and Inferential aspects of validity. We will provide the rationale for and descriptions of each aspect of validity and also describe the forms of evidence that one needs to collect to support a comprehensive validity argument regarding the quality and utility of any given assessment desig-ned to function close to instructional settings.

We will show how existing assessments designed to function close to instruction vary in the extent to which evidence is available for the different aspects of validity and how the nature of the evidence is a function of design features of the specific assessments. The project’s long-range goal is to implement a research- and theory-based framework for the development and use of assessment resources in educational contexts such that well-designed assessments can serve as integral components of the instructional materials and the overall learning environment.

Paper Deducing ‘learning outcomes in action’ from lecturers’ assessment talk.

Susan Orr • Susan BloxhamLearning outcomes are a taken for granted part of the higher education assessment landscape.

In the UK, the national organisation that assures standards and quality in higher education (the QAA) states that institutions need to provide for the ‘effective and appropriate measurement of students’ achievement of intended learning outcomes’ (QAA 2006:31). Given the centrality of lear-ning outcomes it is noteworthy that several studies report that, in spite of QAA guidance, lecturers




50

Paper Developing ‘assessment capable’ beginning secondary teachers in New Zealand.

Frances Edwards • Beverley CooperPre-service teacher educators work with student teachers to help them develop their own

knowledge, beliefs and skills about assessment throughout their teacher education programmes. During their study and practicum experiences student teachers develop a level of assessment capability. This capability is particularly important in the New Zealand context, where secondary teachers are responsible for the design, choice and marking of “internal” assessment tasks used to assess senior students, the results of which contribute directly to the students’ school leaving cre-dential - a standards-based qualification, National Certificate of Educational Achievement (NCEA).

This paper reports on part of a 4-year study which aims to address gaps in our knowledge about secondary conjoint student teachers’ knowledge and beliefs about assessment, and how these change and develop over time. The study involves a systematic investigation of how their ideas and practices develop over the course of a four-year secondary conjoint teacher education pro-gramme.

This was a mixed methods study where students were surveyed yearly, using a questionnaire. Eight students participated in focus group discussions each year, and kept journals for the dura-tion of their teacher education programme.

The presentation will describe findings on the range of views about assessment held by se-condary conjoint pre-service teachers at entry into their teacher education programme and as they move through the programme. Particular factors that contribute to their understandings and practices will be discussed.

Conclusions will be drawn about the implications of the findings and how learning about as-

do not always anchor their assessment decisions exclusively on the basis of assessing student work against the learning outcomes (see for example, Hand and Clewes 2000, Hawe 2003 and Orr 2007). These studies appear to be premised on the expectation that student work would be asses-sed against the documented learning outcomes and they explore the extent to which lecturers’ assessment practices depart from this ideal.

In this study we explore the learning outcomes associated with art and design assessment. However, crucially, we do not anchor this study in the documented learning outcomes found in the course literature. The research reported on here has a different starting point. Art and design lecturers’ assessment talk becomes a key site for carrying out a grounded, bottom up analysis of emergent ‘learning outcomes in action’ used in live assessment contexts. Thus the question is; can we deduce the learning outcomes from the assessment talk of a small group of art and design lecturers?

The authors audio-recorded lecturers while they marked artwork alone (using thinking aloud protocols) and audio-recorded meetings in which lecturers agreed marks in two English universi-ties. The analysis of the transcripts led to the identification of three interrelated learning outcomes in action. These ‘learning outcomes in action’ are used as a lens to explore and challenge the way we understand documented learning outcomes in HE.



51

sessment could be further enhanced over the course of a teacher education programme that involves the simultaneous development of pedagogical and curriculum subject (discipline) knowledge.

Friday 31 augustInvited

Symposium Assessment for learning: varying perspectives with a focus onlearners.

Assessment for Learning (AfL) is a broad concept. In fact, it is often argued that it is ill-defined. Nevertheless it is one of the most active and widespread areas of practice development in as-sessment. The papers in the symposium report on studies using different theoretical perspectives and empirical approaches to examining AfL, bringing clearly-defined frameworks and definitions to bear. Most research on AfL addresses: principles and design; assessment interventions in the curriculum; and impacts of AfL interventions on learning and achievement. A distinctive aspect of the research reported here is the focus on students Although an active learner is central to AfL, the learner perspective has not been well explored. One of the emergent perspectives is that of the student, within a social and institutional context explored using cultural historical activity theory. The concept of assessment literacy is developed in one of the studies, proposing ways in which students need to be prepared if they are to be successful in AfL contexts. As a result, this leads to an assessment curriculum which is centred on long term goals(Boud & Falchikov, 2007). Another paper reports on two practitioner research studies. One identifies the scope and significance of students’ communities of practice beyond the curriculum. A second focuses on students’ construc-tions of AfL noting substantial difference in emphasis when compared to teacher constructions. Students foregrounded dialogue, social interactions and informal feedback rather than specific interventions such training for and participation in self and peer assessment. The implications of the research for practice, particularly in higher and professional education, will be discussed.


Anton HavnesThis paper takes as the starting point that one of the challenges that need to be addressed in

the development of assessment for learning is the theoretical grounding of formative assessment. Research on formative assessment is often associated with constructivism and the notions of in-dependent learning and the self-regulated learner (Sadler 1987, 2010; Nicol, 2009; Wiliam, 2011). However, this emphasis on self-regulation, based on internal control of cognitive processes, is countered by an emphasis on student-teacher dialogue, peer assessment and social interaction

Symposium Chair: Dominique Sluijsmans Symposium Discussant: Liz McDowell

Paper 1.- The role of assessment in learning - A Cultural Historical Activity Theoryperspective.


52

Margaret Price • Berry O’Donovan • Chris Rust • Karen Handley • Birgit den OuterEffective assessment for learning means that students need a more sophisticated understan-

ding of how assessment works and how it relates to their learning. This paper explores the emer-ging concept of assessment literacy to negotiate the changing complexities of assessment in con-temporary higher education. The term ‘literacy’ has traditionally been used in the context of skills and associated with attributes such as fluency, competence, or expertise. However, it also has more subtle connotations relating to ‘gateway’ or ‘threshold’ skills which, once mastered, allow access, not only to further learning and knowledge, but also to academic communities as the custodians of academic standards, practices, customs and culture. On the basis of a programme of empirical research and practice development in assessment for learning, it is proposed that to be fully literate, students need to recognise and understand the many aspects of assessment as well as the relationship between them. It is argued that there is a reiterative relationship between assessment for learning and student assessment literacy, and for current assessment for learning initiatives to take full effect, the development of assessment literacy is needed. It is proposed that Higher Education practice steers away from an ad-hoc, and therefore largely, ineffectual approach to the development of assessment literacy and instead, that students are engaged in a program-me of purposive development activities which have been shown to be effective.

(e.g. Nicol, 2010). There is a mix of a self-regulated learning approach and a more socio-cultural approach building on Vygotsky’s theorising of the role of social interaction in learning and more recent developments in Cultural Historical Activity Theory (CHAT) emphasising the embedded-ness of assessment and learning in social, historical and institutional context. Another challenge is related to the context in which assessment for learning and feedback is framed. In this paper formative assessment will be discussed in the context of some particular challenges in professio-nal education.

I will discuss formative assessment in a CHAT perspective by drawing attention to three levels of analysis that need to be included in cultural historical analysis:

•assessment as component in the teaching-learning relationship, emphasising the role of as-sessment in the zone of proximal development (zpd)•the embeddedness of learning and assessment in an activity system which provides affor-

dances and constraints on educational practices•the interrelatedness between education and the ambient world, in this case professional edu-

cation and professional practice The paper contributes to the discourse of the concept of formative assessment and the relation-

ship between assessment, learning and teaching. It also raised a debate on the particular challen-ges associated with developing formative assessment in professional education.

Kay Sambell • Paul OrsmondIn Assessment for Learning (AfL) the student is regarded as active in the construction of learning

Paper 2.- Assessment literacy - a perspective on the student role in assessment for learning.

Paper 3.- Students’ constructions of assessment for learning: within and beyond the cur-riculum.


53

and assessment, so it is important that the student perspective and experience is researched. Two close-up investigations of university students’ constructions and practices in relation to AfL are reported. Both studies reveal aspects which are often neglected when AfL is viewed solely as a teacher-led, curriculum-led practice. It is identified that students create their own ‘hidden curricu-lum’ in response to assessment for learning. This is not a negative phenomenon but builds on the roles that we expect learners to play. One of the empirical studies used the conceptual framework of communities of practice. It revealed the extent and quality of student communities of practi-ce which are largely overlooked in favour of promoting participation in academic communities. Student communities of varying kinds, established quite independently of teachers, were shown to fulfil functions including: sharing exemplars of academic work, testing knowledge, motivation and developing identities. In interaction with students, tutors should be aware that they are not just speaking to individuals but to another community that has its own rules and practices. In another empirical study, students produced guides to assessment for learning for a student au-dience. Content analysis of students’ guides showed that they tended to interpret and reframe the assessment for learning agenda in different ways and with noticeably different emphases than the ones which are prominent in the texts written for and by academics. They placed markedly heavier emphasis on student participation in teaching, learning and feedback practices in general and particularly foregrounded dialogue, social interactions and informal feedback. They remained relatively silent about interventions designed to build students’ appraisal experiences. Implica-tions for assessment practice are discussed.

Symposium Teacher and student roles during feedback in interaction.

In education it is common practice that students have feedback dialogues with their supervi-sors or teachers. A feedback dialogue is a conversation in which students receive verbal feedback from their teacher and exchange information about their performances and learning process and performances with the teacher in order to reduce the gap between their performances and the intended learning goals. Being able to reduce ‘the gap’ requires that students: a) have a concept of the standard of performance, comparable to that of the teacher; (b) continuously compare the actual level of performance with the standard, which implies reflective thinking activities like analyzing and evaluating;(c) engage in appropriate actions to close the gap, which means that students have a repertoire of suitable strategies to draw from (Sadler, 2010). In such a conversation feedback can be seen from an interpersonal (social), cognitive and affective perspective. In this symposium these perspectives are key to research oral feedback situations between teachers and students in secondary education, higher education and vocational education. The symposium encloses four papers: 1) A position paper that discusses the value of the feedback concept from a positive psychological, interpersonal and affective perspective (Voerman et al.); 2) A research paper regarding interaction in feedback dialogues to stimulate students’ reflective thinking (Van der Schaaf); 3) A research paper about the relation between teacher-student interpersonal beha-vior and characteristics of feedback (De Kleijn et al.); 4) A research paper in which an instrument is

Symposium Chair: Marieke van der Schaaf Symposium Discussant: D. Carless9h-10h30: Breakout room 1 Euro


54

Lia Voerman • Paulien C. Meijer • Fred Korthagen • Robert Jan SimonsIn this paper we first present a brief summary of the theory on teacher feedback. Than we intro-

duce some issues that until now seem to have been overlooked or undervalued in most literature on feedback. We add the concept of progress feedback to the conception of feedback as feedback on the discrepancy of an initial level of performance and a goal. Progress feedback is described as feedback comparing an initial level of performance with a current level of performance. We make a distinction between praise and feedback on the self and describe both concepts separately. Praise is described as non specific positive feedback. Building on research result from positive psychologists we propose to provide feedback on character strengths as a way to provide fee-dback on the self. Research form positive psychologists showed the impact of emotion on lear-ning. Elaborating on these research findings, we discuss the impact that feedback has on emotion and thus on learning. Hence we stress the importance of taking into account the emotion that is evoked by feedback, while providing feedback. Implications for teaching practice in schools entail recommendations for teachers how to provide effective feedback.

Marieke van der Schaaf How can feedback dialogues stimulate students’ reflective thinking? This study aims to inves-

tigate: (1) the effects of feedback dialogues between teachers and students on students’ percep-tions of teacher feedback and (2) the relation between features of feedback dialogues and stu-dents’ thinking activities as part of reflective thinking. A quasi-experiment was carried out in which seventy-two secondary education students received written teacher feedback. Half of them had an additional feedback dialogue with their teacher. The latter group perceived teachers’ feedback as more useful than the group without a dialogue. The feedback dialogues could be clustered in two groups that varied in the number of segments containing teacher and student interaction. The number of segments containing interaction was positively correlated with students’ use of thinking activities.

constructed to measure students’ interpersonal behavior within teacher-student feedback situa-tions (Prins & Mainhard).

Renske de Kleijn • Paulien Meijer • Albert Pilot • Mieke BrekelmansIn master’s thesis supervision one of the most important tasks of supervisors is to provide cons-

tructive feedback, often in feedback dialogues. As previous research has indicated the importance of a feedback perceptions and an interpersonal perspective on these feedback dialogues, both perspectives are investigated in the present study: how are student interpersonal and feedback perceptions related and to what extent can a combination predict student satisfaction (SS) and perceived supervisor contribution to learning (PSCL)?

Paper 1.- Some issues on teacher feedback.

Paper 2.- Interaction in Feedback Dialogues to Stimulate Students’ Reflective Thinking.

Paper 3.- Interpersonal and feedback perceptions in master’s thesis supervision.


55

The results of this study will provide insight in the relation between two perspectives on fee-dback dialogues (feedback and interpersonal perceptions) and will yield insight concerning to what extent these perspectives overlap and/or explain distinct variance in SS and PSCL. Feedback perspective

Based on the findings of two review studies concerning feedback (Hattie & Timperley, 2007; Shute, 2007) feedback perceptions are conceptualized in terms of perceptions of focus, elabora-tion, and goal-relatedness.Interpersonal perspective

Perceptions of the supervisor interpersonal behavior were conceptualized based on the model for interpersonal teacher behavior in which the two interpersonal dimensions control and affilia-tion are discerned.

In total 1016 master students completed the questionnaires.Instruments:

•Feedback perceptions: QFP (de Kleijn et al., submitted)•Interpersonal perceptions: QSDI (Mainhard et al., 2009)•SS: two items (r=0.81; de Kleijn et al., in press)•PSCL: five items (α=0.89; de Kleijn et al., in press)

Correlation analyses, cluster analysis and ANOVAs and regression analyses were applied.The results of this study suggest that both a feedback perceptions perspective and an interper-

sonal perspective on feedback dialogues in master’s thesis supervision provide partly overlapping but also distinct information as it was found that all subscales are related, but that also all scales except for focus were significant predictors for SS and PSCL. Affiliation was found to be the most strong predictor.

Frans Prins • Tim MainhardNicol (2010) and Carless et al. (2011) argue that feedback can be more effective when it is pro-

vided during a dialogue between teacher and student instead of by monologues from teacher to student. During a feedback dialogue, teacher and student may adopt different roles (e.g., Hadwin et al., 2005) and they may adapt their behavior to the behavior of the other person. Such inter-personal dynamics of feedback dialogues have been shown to be related to the effectiveness of feedback. It is therefore important to examine how interpersonal teacher behavior and interper-sonal student behavior during feedback dialogues affect each other. However, there is no instru-ment available to measure supervisors´ interpersonal perceptions of student behavior. This paper describes the construction of a questionnaire that taps supervisors´ interpersonal perceptions of student behavior during feedback dialogues.

Two studies are presented. Study 1 is a Delphi study in which 10 experts rated descriptions of student behavior in feedback dialogues, provided by students and experienced teachers, on sui-tability and interpersonal meaning. The experts also categorized these descriptions on the inter-personal circle. Study 1 resulted in a prototype of a questionnaire on interpersonal perceptions of student behavior in teacher-student feedback dialogues. In study 2, the questionnaire was tested on reliability and validity. Results of study 2 indicated that the questionnaire is a reliable instru-

Paper 4.- Tapping supervisors´ interpersonal perception of student behavior during feedback dialogues: development of a questionnaire.


56

ment, but that the reliability of items representing passive student behavior was somewhat lower. Furthermore, interpersonal profiles of, according to supervisors, good, average and ineffective students differed significantly, which indicates the external validity of the questionnaire. Finally, the circumplex structure of the items and scales of the questionnaire indicated internal validity. Thus, the questionnaire in its current form is a reliable and valid instrument that can be used for re-search on the interpersonal aspects of the interplay/dynamics between teacher/supervisors and student during feedback dialogues.

Paper Comparing different item-types in paper-based and computerized environment.

László HülberAltering the measuring valuation processes of technologically based results is not only modi-

fying the medium but the CBA provides such possibilities - compared to the PB testing which can

Paper Innovative Assessment Technologies in Educational Games Designed for Integrating Assessment into Teaching.

Gyongyver Molnar • Brigitta Mikszai-Rethey • Attila Pasztor • Timea MagyarTechnology offers a great deal of new opportunities in educational assessment and vitalizes

testing situation by allowing innovative task presentation, including multimedia, which increa-ses motivation. Instead of providing single indicators, such as a test score, technology-based as-sessment may produce rich, contextualized data sets, which supports the transition from single testing to complex systems of assessments. The purpose of this pilot study is to investigate the opportunities and effectiveness of applying educational games and embedded innovative as-sessment technologies to improve students’ reasoning skills. First and second grade (age 6 to 8) students constituted the experimental group (n=38), whereas the control group consisted of 137 students. The effectiveness of the training was measured with an inductive reasoning test, compri-sing 37 figural, non-verbal items (Cronbach α=.87). Beside test-based data collection, innovative assessment technologies are explored by logging metadata, such as facial expressions and head movement (captured by web camera). There was no significant change in performance between pre- and post-test in the control group (t=1.44, p>.05), while the experimental group managed to achieve significant development in the experimental period (t=-10.65, p<.00), significantly out-performing the control group by more than one standard deviation on the post-test. The effect size of the training program was d=.87 (p<.01). The most frequent facial expressions were those of surprise (31%), happiness (24%) and anger (16%) during the training. Disgust (7%), fear (3%) and sadness (1%) were less frequent. No significant differences were found between genders with regard to the distribution of the different expressed emotions. The distributions of the facial ex-pressions did not show significant relationships with level of reasoning skills. Our findings pro-vided further empirical evidence that innovative assessment technology gives the possibility to integrate assessment into teaching by giving proper assessment tools to monitor both cognitive (e.g. reasoning) and affective (motivation, interest, boredom) processes.




57

reform the whole measuring valuation process. For responsible alteration the media effect has to be identified with regards of (1) sample, (2) test characteristics, (3) technical parameters. My ob-jective was to separate those mathematic erudition fielded items which are primarily responsible for the CBA and PP result discrepancy, to collect their mutual characteristics and to type them. I have used the diagnostic measurement data recorded in 2010 in Hungary on CBA (N=943) and PP (N=20400), the testing carried out in the first and sixth grades. I have used those 299 items for item diagnostics, which had already been defined on both the CBA and PP. I started the data analysis with item response theory models, which gave the opportunity to characterize those students who are in different grades but filled out tests based on mutual ability scale, furthermore, to cha-racterize items in different tests on the same scale. I made further analysis with the help of classical test theory methods on the result of the multidimensional analysis items. The results show that the item difficulty correlation in CBA and PP is r= .64, which is significant on the level of p< .01. The item difficulty values are weakly correlating with the item’s character number which are at CBA: r= .32 p< .01, at PP: r= .25 p< .01. There is no significant difference neither at the open ended tests in the PP and CBA results: t=1.18, p= .24; nor at the closed ended tests t=-1.07, p= .28. With applying the helpful or needed graphical elements, there is a significant difference between the CBA and PP results: t=2.22 p< .05.

PaperAssessment Planning for Interactive Case-based Learning Scenario.

Genevieve Gauthier • Jeannine Conway • Rebecca TaylorThe use of cases in teaching offers the potential to meet the challenges of today’s evolving

knowledge, yet the evaluation of case performance poses challenges to current assessment prac-tices. This research explores how expert instructors conceive and plan assessment through the design of interactive computerized cases in pharmacy education. This initial study we explore instructors’ pre-assessment planning for cases they will be using with their students. We describe and discuss reliability issues regarding setting proposed “good” answers for problem solving tasks. Assessment challenges were raised during the development of patient cases for an interactive computerized program called BioWorld. When experts and case creators were asked to perform relatively easy cases using the software, we encountered reliability issues regarding the proposed “good” answers for the cases and the answers the experts actually performed. Experts and case creators could neither reproduce the “proposed” good answers for the cases, nor upon replica-tion of the cases, repeat their first answer to the cases. We suggest that assessment of case-based learning is more complex than simply identifying the “right” answer to a problem and it may be useful to conceptualize the notion of case solving as an ill-defined problem solving performance, as opposed to simply a way to trigger knowledge.



58

PaperHow multiple perspectives complicate comparability.

Gill ElliotMultiple perspectives are at the heart of comparability issues and dilemmas. Leaving aside the

issue of comparability over time, if students are permitted to make choices in education, there will be a need for comparability between the different options which are available. In the English edu-cation system there exist high stakes assessments taken by many students at the ages of sixteen and eighteen years. Students have many choices, both about which subjects they take and which particular assessments they choose. The choices they make can impact upon their opportunities for Higher Education entry or employment. An effective means of making comparisons between these different pathways needs to be applied to ensure fairness in a situation where increasing numbers of competing students offer a wide variety of different qualifications in a large number of subjects.

In this paper the author discusses two separate issues relating to multiple perspectives which affect comparability of large scale assessments in England. Firstly, the multiple dimensions that constitute comparability will be explored, and the impact that these dimensions have upon the public understanding of the issue and the effectiveness of solutions which have been attempted will be discussed. Secondly, the diverse perspectives of the different users of the assessments will be investigated, revealing that comparability is not necessarily defined in the same way in all si-tuations.

The way in which comparability is perceived affects both educational policy and practice in England. In drawing conclusions about the effect of multiple perspectives upon the implementa-tion of comparability practice the author will consider some of the evidence from the past as well as the direction of policy in the future.

Paper First results of Hungary´s full cohort longitudinally connectedassessment.

Ildikó BalázsiHungary’s National Assessment of Basic Competencies, an annual full cohort mathematics and

reading literacy assessment, with its latest innovation, the introduction of the measurement ID in 2008, became a longitudinal study. The paper explains the National ABC‘s main features. First, the assessment frameworks and test administration processes are shortly introduced. The scaling and analysis methodology is discussed, the grade- and year-invariant IRT scales and profession levels, weighting and computing algorithms of error terms are described. Then, the paper focuses on the newly developed longitudinal aspects, the analysis of the two year long development of students’ abilities on the measured cognitive domains.

The paper briefly presents the main results of the 2010 and 2011 assessments, focusing on the longitudinal results, the development of students’ abilities during the two years between two assessment points. The relationships of settlement type, school type and socio-economic status




59

PaperDeveloping a coding scheme for analysing peer feedback messages.

Jan-Willem Strijbos • Bas van Goozen • Frans PrinsIn line with the increased attention for peer assessment research, the peer feedback component

of peer assessment is also more systematically considered. Recent studies that examined the con-tent of peer feedback revealed that the content of peer feedback appears to be a crucial aspect for its effectiveness. However, although several studies analysed the content of peer feedback messages, the coding schemes developed are not systematically tied to a theoretical framework and almost exclusively address cognitive processes. Hence, this study reports the development of a coding scheme for peer feedback messages that moves beyond the cognitive dimension. In addition, the coding scheme heavily draws on the feedback framework developed by Narciss (2008) and distinguishes between ‘feedback function’ and ‘feedback content’. The developed co-ding scheme consisted of five main categories (cognition, meta-cognition, affect, self-efficacy and noncodable) and 24 sub-categories. Cognitive and meta-cognitive processes were further refined with sub-categories for positive and negative verification and five types of elaboration (question, correction, confirmation, suggestion and justification). Since the coding scheme heavily draws on the framework by Narciss (2008) it can be considered theoretically valid. Reliability trails revealed that the coding scheme is sufficiently reliable at the main-category level (cognitive, meta-cogni-tive, affective, self-efficacy and noncodable), as well as the distinction between verification and elaboration. At the sub-category level the reliability is satisfactory, but could be further enhanced in future studies. The coding scheme developed allows for more precise analyses of peer feedback messages and investigating and determining which aspects of a peer feedback message affect the application of peer feedback.

with the results and the development are dealt with. Finally, report types and reported content are discussed: such as online schools’ and maintai-

ners’ reports that provide a multi-aspect review of the results compared to national results and to the results of similar schools in regard the schools type, student intake and initial abilities of students as measured in the two year earlier assessment.

Paper Prompt Feedback by an assessment instrument (Evalcomix) in an Engineering Degree

Daniel Garcia-Almiñana • César Valderrama • Beatriz Amante • María MartínezThe adapting process to the European Higher Education Area (EHEA) gives a different value to

the evaluation process, since it has not only be a certification of that students have learned, but the overall process in which the students can be aware of how and what is learned. This implies that the assessment should be formative as well as summative, for instance, it must be focused to promote or facilitate learning in addition to a normal scoring process. A fast and detailed feedback of learning process is required and allows students to continuous improvement. The teacher fee-




60

dback is not always required; but it is recommended that the students themselves or their part-ners also can offer this feedback. Thus, feedback can be very useful both to motivate and involve students in their own learning process and to accounting for non-lecture time of the students; on the other hand, this may reduce the teaching load especially for large groups. In order to assure that students participation in feedback helps and influences the improving learning process is es-sential to define with clarity the criteria and the objectives of assessment tasks; furthermore they should be used easily and automatically in order to ensure the immediacy of the feedback. One of the main objectives of The GRAPA innovation group at the Universitat Politècnica de Catalunya is provide to the teachers of strategies and assessment tools to facilitate the adaptation to the EHEA. EvalCOMIX is one of these useful tools which have been developed to facilitate feedback in assessment process under compatible structure to the virtual campus (Moodle).

In this work is reported the assessment tools created and used through the EvalCOMIX in the subject of Engineering Projects of Engineering in Industrial Organisation degree (part-time atten-dance course) during the course of 2011-2012.

Kristine Høeg Karlsen • Joakim KarlsenThis paper reports from our work on understanding the complex relationship between types of

teacher feedback and students learning activities. We have applied Karlsen’s (2011; in press) latest work on classification of feedback-types and Krathwohl’s (2002) revisited framework of Bloom’s taxonomy of educational objectives in order to explore this relationship. A range of factors in-fluences what kinds of learning activities students engage in. To single out and investigate the causal relationship between these activities and teacher feedback is challenging. In our research we don’t claim any direct causal relationship between the phenomena, but try to analyse and un-derstand students and teacher’s language and actions together framed by short feedback loops. Findings are based on the analysis of these texts internal references. The teacher’s and the stu-dents in this study frequently refer to specific statements, tasks and topics occurring in the same feedback-loop.

A qualitative approach using ethnographic methods for data collection and analysis was used. Two comparative cases at two faculties in the field of technology at the same university college in Norway were chosen. In both cases students was working on their bachelor thesis group-projects, similar in both content and organisation. The sample consists of 28 students and their six super-visors. The collected data material consist of; 97 field observations, including observation of 25 whole feedback-loops , 28 semi-structured one-to-one interviews with students and approxima-tely 400 web-based e-logs written by the students.

Preliminary analysis seems to give strengths to our assumption that for teacher feedback to be used by these undergraduate students it has to be well-styled and its form needs to be adapted to the type of learning activities (remembering, understanding, applying, analysing, evaluating


Feedback matters when students are considering what type of learn-ing activities to engage in - exploring the relationship between feed-back-types and students learning activities in higher education.

Paper


61

Paper Perceptions of the purposes of portfolio assessment in primaryeducation.

Diana Baas • Mien Segers • Jos Castelijns • Marjan Vermeulen • Rob MartensNowadays, portfolios are becoming increasingly popular in primary education as a tool to pro-

mote learning. In spite of the popularity of portfolios, different definitions are used and a wide array of purposes is being served, ranging from summative to formative. In addition to various purposes of portfolio assessment, different stakeholders are involved in the portfolio process in primary education. However, when stakeholders have differing perceptions of the purpose of the portfolio, the power of the portfolio as a learning tool can become compromised. This study fo-cuses on stakeholders’ perceptions of the purposes of portfolio assessment in primary education. Furthermore is being studied whether perceptions of the purposes of stakeholders differ. Semi-structured interviews with principals, teachers, students and parents were conducted in ten pri-mary schools. Validated summaries of the interviews were coded with MAXQDA using a grounded theory approach. Results show that stakeholders perceive the purposes of portfolio assessment in primary education to be primarily formative. Portfolios are perceived as a powerful tool to give students responsibility in their own learning process, to monitor their learning and to reflect on learning. Concerning these purposes, no differences in perceptions of the various stakeholder groups could be found. However, stakeholders reported differing perceptions on the use of port-folio assessment for communication and feedback purposes. With respect to these purposes the-re is still room for improvement.

Paper Primary school teachers’ use of classroom assessment in Dutchmathematics education.

M. Veldhuis • M. van den Heuvel-Panhuizen • J. A. Vermeulen • T. EggenIn this study we investigate the classroom assessment practices of Dutch primary school tea-

chers and their view on the role of assessment in their teaching practice. We consider classroom assessment the process of collecting data by teachers about their students’ learning process in such way that teachers can make informed decisions about how to continue in their teaching. From previous research is known that teachers can use of a wide range of assessment methods and that teachers’ use of assessment methods is related to teachers’ assessment purposes and tea-chers’ beliefs concerning classroom assessment. However, despite the use of formative assessment, summative assessment practices have still been found to dominate educational policies of most governments. As a result teachers may perceive assessment aimed at assessing students’ edu-cational needs as less important, which in turn may result in unbalanced classroom assessment practices. Unclear is what this means for the teachers’ classroom assessment practice in mathema-tics education in Dutch primary schools. This study aims at getting a more clear picture of this, and exploring how several classroom assessment aspects are related on the teacher level as well on


and creating) the students are willing to engage in at that point of time in their learning processes.



62

PaperDynamic cognitive assessment of young adults.

Alex KozulinObservation that students’ independent task performance do not necessarily coincide with their

learning potential or cognitive modifiability prompted Vygotsky to formulate the concept of the Zone of Proximal Development (ZPD). Later on the concept of ZPD became an inspiration for so-called dynamic assessment (DA) procedures that focus on students’ ability to benefit from a short mediated learning experience provided during the assessment. The majority of DA studies aimed at demonstrating that DA helps minority children and children with special needs to reveal their “hidden” potential. Many aspects of DA, however, remained unexplored particularly in application to adult learners with typical development: 1) What is the range of learning potential of adult learners who have the same unassisted performance scores; 2) Does learning potential constitute a general ability, or is it modular, dependent on DA task content and modality; 3) Is there a diffe-rence between learning potential demonstrated during DA and cognitive change in response to a long-term educational intervention. These questions were addressed in two studies with large groups of young adults. The results indicate that subjects with the same static test scores have widely different learning potential; learning potential appears to be modular, dependent on the type of DA tasks; learning potential and cognitive modifiability seem to reflect different abilities.

the school level. Data are collected through an online teacher questionnaire. This questionnaire consisted of a number of questions about relevant teacher background characteristics, teachers’ use of assessment methods and teachers’ beliefs about and views on the role of assessment. After conducting a non-response study, we will first use univariate descriptive statistics to describe the classroom assessment practice and teacher and school characteristics, furthermore correlation analyses will be used to find out how assessment methods, uses, and purposes are related (on the school and teacher level), and finally (multi-level) regression analyses to measure the influence of teachers’ beliefs and characteristics on their use of assessment methods.

Paper How do university teachers assess their students? Understand assess-ment from their own viewpoint.

V. Quesada-Serra • G. Rodríguez Gómez • Mª S. Ibarra Sáiz • E. García JiménezRecent tendencies regarding assessment in higher education such as the ones proposed by

Boud (2010), Carless, Joughin and Mok (2006), Falchikov (2005) or Nicol (2010) highlight the need to implement alternative assessment practices to enhance students learning that take into ac-count aspects like: develop authentic assessment tasks, provide quality feedback and feedforward to students and promote student’s active participation in the assessment process. A survey re-search method was carried out with the aim to provide the opinion of university teachers regar-ding their assessment activity, with regards to the importance, competence and use of thirty-one




63

Paper Assessment literacy in university students: what is it and how is it developed?

Birgit Den OuterTo perform well students must master the material of their chosen subjects but also need to

develop an expertise to help them negotiate assessments as members of a learning community. In this study, the latter requirement is referred to as ‘assessment literacy’, the research into which is approached holistically with the distinction between formative and summative assessment de-liberately blurred. The paper is a presentation of the findings of a two-year project among un-dergraduate and postgraduate students. Central questions are: how do students develop their assessment literacy? What place do assessments take in students’ perceptions of themselves and how do they use assessment experiences of the past to negotiate assessments of the future? Au-dio diaries are used to track experiences of 20 students from several disciplines at a post-1992 university. In addition, data derived from semi-structured interviews further illuminate aspects of these experiences by situating them in respondents’ broader educational (hi(s)tories. An initial conceptualisation of assessment literacy draws on academic literacies models, and educational narrative and developmental identity literatures. Findings to date point towards two propositions: firstly, students use assessment experiences to construct, reinforce, or challenge their narrated self in the domain of learning through which they develop their assessment literacy. Secondly, a developed assessment literacy is where learning from feedback, either summative or formative, is aligned with a narrated self. The paper will illustrate these two propositions with material from the data and discuss the circumstances in which certain types of assessments supported or undermi-ned the research participants’ learner identity and how this is linked to wider, socially constructed notions on success and failure in the learning environment.

assessment tasks previously selected. The sample comprised 427 university teachers from 18 Spa-nish universities; most of them had over ten years of teaching experience. The results show that teachers gave higher scores to those assessment tasks related to ‘assessment planning and design’ and ‘monitoring of student’s learning’. On the contrary, the most innovative practices such as the items related to promoting the ‘participation of students in assessment’ received the lowest scores of the questionnaire -Primarily regarding the ability to put those assessment tasks into practice and the extent to which they already use some of them in their classes-. In general, university teachers considered most of the tasks as very important, the competence and the use of those assessment tasks are, however, lower. Therefore, we can conclude that university teachers see the importance of another type of assessment rather than the traditional. Higher education institu-tions must take part and promote policies that enhance alternative assessment practices. To do so, it is essential that university teachers feel confident and capable enough to introduce them.



64EARLI SIG 1ASSESSMENT AND EVALUATION

Documents

EARLI SIG 1: CONFERENCE PROGRAM