Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
1EARLI SIG 1ASSESSMENT AND EVALUATION
CO N F E R E N C E P R O G R A M
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
Linking Multiple Perspectives on
Assessment
28-31 August, 2012 Brussels
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
2
SIG 1 COORDINATORSDominique Sluijsmans
Katrien StruyvenErnesto Panadero
CONFERENCE PROGRAMSixth biennial meeting of Earli Sig 1: Assessment and Evaluation
"Linking Multiple perspectives on Assessment"
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
3
Mai
n co
nfer
ence
room
Oly
mpi
a Bo
urse
Ba
llroo
m
Brea
kout
room
1
Euro
Brea
kout
room
1
Ster
ling
Brea
kout
room
1
Dol
lar
Brea
kout
room
1
Yen
10h30
Ope
ning
11h-12h30
Keyn
ote
(p.6)
Jose
ph K
esse
ls12h30-
14hRe
stau
rant
Mid
twon
Gril
l
14h- 15h30
Sym
posi
um (p
.6)D
avid
Bou
dRe
fram
ing
feed
back
: ne
w p
ersp
ectiv
es, n
ew
dim
ensi
ons.
Jenn
y Le
nkei
t (p.9
)H
ow e
ffect
ive
are
educ
atio
nal
syst
ems?
A v
alue
-add
ed a
ppro
ach
to st
udy
tren
ds in
PIR
LS.
Khal
ed B
arka
oui (
p.10)
Effec
ts o
f Key
boar
ding
Ski
lls
on P
erfo
rman
ce o
n Co
mpu
ter-
Base
d L2
Writ
ing
Test
s.
G. B
row
n an
d A
. Rem
esal
(p.12
)Sp
anis
h as
For
eign
Lan
guag
e Te
ache
rs’ c
once
ptio
ns o
f as
sess
men
t: pr
elim
inar
y re
sults
from
an
inte
rnet
inqu
iry
Jose
ph C
onbo
y (p
.14)
Teac
her C
omm
unic
atio
n of
Ev
alua
tion
Resu
lts: I
mpa
ct
on S
tude
nt E
ngag
emen
t in
Sch
ool.
Dan
iel C
aro
(p.9)
A N
ew Lo
ok a
t the
Eva
luat
ion
of S
ocio
logi
cal T
heor
ies i
n In
tern
atio
nal L
arge
Sca
le
Educ
atio
nal A
sses
smen
ts.
Ast
rid
Birg
itte
Egg
en (p
.11)
Dev
elop
ing
natio
nal s
tand
ards
fo
r the
ass
essm
ent o
f writ
ing
in
com
puls
ory
educ
atio
n.
Gav
in B
row
n (p
.13)
Asse
ssm
ent i
n Br
azili
an h
ighe
r ed
ucat
ion:
Val
idat
ion
of a
Po
rtug
uese
ver
sion
of t
he S
tude
nts’
Conc
eptio
ns o
f Ass
essm
ent (
SCoA
-VI
) inv
ento
ry.
Robi
n Ti
erne
y (p
.15)
Thin
king
abo
ut F
airn
ess i
n Cl
assr
oom
Ass
essm
ent.
Alis
on G
ilmor
e (p
.10)
Revi
sion
ing
Nat
iona
l Ass
essm
ent
in N
ew Z
eala
nd.
Khal
ed B
arka
oui (
p.12)
Exam
inin
g Ta
sk E
quiv
alen
ce
in L
2 W
ritin
g Te
sts:
A M
ixed
M
etho
ds A
ppro
ach.
Wei
Shi
n Le
ong
(p.14
)Co
ncep
tions
of C
lass
room
As
sess
men
t: Fi
ndin
gs fr
om U
sing
Q
-Met
hodo
logy
.
Caro
l Eva
ns (p
.16)
The
emot
iona
l dim
ensi
on
of fe
edba
ck.
15h30-16h
Coffe
cor
ner
16h-17h30
Roun
dtab
le (p
.16)
Mai
n co
nfer
ence
room
Oly
mpi
a Bo
urse
Bal
lroom
Wor
ksho
p 1
(p.25
)G
odfr
ey P
ell,
and
Rich
ard
Fulle
rTh
e ch
alle
nge
of m
eetin
g th
e as
sess
men
t nee
ds o
f bot
h co
mpe
tent
and
und
erpe
rfor
min
g st
uden
ts w
ithin
exi
stin
g ed
ucat
iona
l str
uctu
res:
Intr
oduc
ing
sequ
entia
l tes
ting?
Le
sson
s for
pol
icy
and
prac
tice.
Wor
ksho
p 2
(p.26
)Ed
uard
o G
arcí
a Ji
mén
ez,
Mª S
oled
ad Ib
arra
Sái
z an
d G
rego
rio R
odríg
uez
Góm
ezEv
alCO
MIX
: A w
eb se
rvic
e fo
r u-
feed
back
and
u-fe
edfo
rwar
d.
Wor
ksho
p 3
(p.26
)N
iek
van
Bent
hum
A th
eory
of i
mpr
ovem
ent f
or
teac
hers
’ pro
fess
iona
l dev
elop
men
t in
Ass
essm
ent f
or Le
arni
ng
Gav
in B
row
n. (p
.16) A
ccur
acy
in S
elf-
asse
ssm
ent:
Deb
atin
g th
e M
erit
of A
ccur
acy
whe
n St
uden
ts E
valu
ate
thei
r Ow
n W
ork.
Pa
tric
k La
i. (p
.17) I
ntro
duct
ion
of q
ualit
y as
sura
nce
elem
ents
into
the
asse
ssm
ent p
olic
y of
gen
eric
gra
duat
e at
trib
utes
of u
nive
rsity
usi
ng th
e co
urse
-em
bedd
ed a
sses
smen
t m
etho
ds.
Hui
yong
Sea
h-Ta
y. (p
.18) W
hat T
ypes
of A
sses
smen
t Con
text
s Sup
port
Sel
f-Reg
ulat
ed Le
arni
ng?
Kare
l Van
Nie
uwen
huys
e. (p
.18) T
he re
latio
nshi
p be
twee
n pa
st a
nd p
rese
nt in
end
of t
erm
test
s in
Flem
ish
hist
ory
educ
atio
n.M
adda
lena
Tar
as. (
p.19)
Und
erst
andi
ng o
f Ass
essm
ent b
y U
nive
rsity
Lect
urer
s: ho
w th
is li
nks t
o th
eorie
s.A
mir
a Ro
m a
nd M
icha
l Nac
hsho
n. (p
.20) H
igh
Scho
ol S
tude
nts P
ortfo
lios i
n Sc
ienc
e fo
r All:
Indi
cato
rs o
f the
Qua
lity
of th
eir T
each
ers W
ork.
Núr
ia d
e Sa
lvad
or. (
p.20)
Form
ativ
e As
sess
men
t usi
ng C
olle
ctiv
e Po
rtfo
lios.
Mas
ahir
o A
rim
oto.
(p.21
) Lin
king
a fr
amew
ork
of fo
rmat
ive
asse
ssm
ent a
nd e
valu
atio
n fo
r “le
arni
ng to
lear
n” to
lear
ning
circ
les f
or tr
ansf
orm
atio
nal s
choo
l lea
ders
hip.
Rita
Hea
ding
ton.
(p.21
) Exp
lorin
g un
derg
radu
ate
stud
ents
’ inf
orm
al, f
orm
ativ
e pe
er a
sses
smen
t net
wor
ks .
Ho
Yi X
ian.
(p.22
) Stu
dent
s’ un
derp
erfo
rman
ce d
ue to
pse
udo
unde
rsta
ndin
g of
mat
hem
atic
al co
ncep
ts: W
hat c
an b
e im
prov
ed in
scho
ol a
sses
smen
t?St
even
Nijh
uis.
(p.23
) Will
dig
itally
aid
ing
the
crea
tion
of a
pro
ject
pla
n cr
eate
tran
sfer
?Kr
isti
en C
arne
l. (p
.23) U
ITZI
GT
inte
grat
ed a
sses
smen
t of c
ompe
tenc
es o
n ou
tflow
leve
l in
teac
her e
duca
tion.
Chri
stin
e Ca
rolin
e Jä
hnig
. (p.2
4) A
sses
sing
pro
cedu
ral K
now
ledg
e of
Ger
man
Bus
ines
s Stu
dent
s.M
icha
el B
ots.
(p.24
) Aut
hent
icity
and
por
tfolio
ass
essm
ent.
Wednesday 29 august
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
4
Mai
n co
nfer
ence
room
Oly
mpi
a Bo
urse
Ba
llroo
m
Brea
kout
room
1
Euro
Brea
kout
room
1
Ster
ling
Brea
kout
room
1
Dol
lar
Brea
kout
room
1
Yen
9h-10h30
Invi
ted
Sym
posi
um (p
.27)
Shar
on K
linke
nber
gIn
vite
d Sy
mpo
sium
SIG
-DT,
Th
eme:
Test
-enh
ance
d le
arni
ng m
eets
lear
ning
an
alyt
ics.
Sym
posi
um (p
.28)
Lies
je C
oert
jens
Stud
ent a
ppro
ache
s to
lear
ning
and
thei
r pe
rfor
man
ce: E
vide
nce
from
prim
ary
and
high
er
educ
atio
n.
10h30-11h
Coffe
cor
ner
11h00-12h30
Sym
posi
um (p
.30)
Sue
Blox
ham
Asse
ssm
ent:
safe
in o
ur
hand
s?
Jan
Ard
ies
(p.33
)D
evel
opm
ent o
f an
Asse
ssm
ent T
echn
olog
ical
Li
tera
cy.
E. P
anad
ero
(p.35
)Fr
iend
ship
in th
e re
liabi
lity
of n
on-r
ubric
an
d ru
bric
-bas
ed p
eer
asse
ssm
ent.
Caro
lin R
amst
eck
and
Ann
ette
Fru
ehw
acht
(p.37
)A
Feed
back
Inte
rven
tion
Theo
ry
Pers
pect
ive
on E
xter
nal S
choo
l pe
rfor
man
ce D
ata
Use
in
Ger
man
Sec
onda
ry C
lass
room
s.
Sasc
ha W
üste
nber
g (p
.38)
Com
plex
Pro
blem
Sol
ving
in
Educ
atio
nal C
onte
xts –
Asse
ssm
ent,
Mea
sure
men
t Inv
aria
nce,
and
Co
nstr
uct V
alid
ity.
Carm
en T
omas
(p.33
)In
tegr
atio
n of
e-a
sses
smen
t pr
oces
ses f
or la
rge-
scal
e in
stitu
tiona
l dev
elop
men
t.
Robb
ert S
mit
(p.35
)Ru
bric
s as a
link
be
twee
n st
anda
rds a
nd
form
ativ
e as
sess
men
t.
Chan
g Zh
u (p
.37)
Prov
idin
g fo
rmat
ive
feed
back
to
stud
ents
via
em
ails
and
fe
edba
ck st
rate
gies
rela
ted
to
stud
ent m
etac
ogni
tion.
Gor
don
Joug
hin
(p.39
)M
ultip
le p
ersp
ectiv
es o
n a
viva
voc
e as
fina
l ass
essm
ent i
n ve
terin
ary
scie
nce.
Kris
Buy
se (p
.34)
Effec
ts o
f str
ong
port
folio
tr
igge
rs o
f heu
ristic
in
stru
men
ts a
nd li
nked
su
mm
ativ
e/fo
rmat
ive
asse
ssm
ent o
n w
ritin
g,
writ
ers a
nd tu
tors
.
And
ers
Jons
son
(p.36
)Sc
orin
g ru
bric
s for
st
uden
t lea
rnin
g.
Sara
h M
alon
e (p
.38)
Com
pute
r bas
ed a
sses
smen
t of
driv
ing
com
pete
nce
invo
lvin
g di
ffere
nt re
spon
se m
odes
Céci
lle R
aves
loot
(p.40
)Q
ualit
y im
prov
emen
t of s
umm
ativ
e ra
diol
ogic
al sk
ills a
sses
smen
t with
di
gita
l mul
ti-sl
ice
imag
es in
med
ical
ed
ucat
ion.
12h30-14h
Rest
aura
nt
Mid
twon
Gril
l
14h- 15h30
Keyn
ote
(p.40
)M
enuc
ha B
irenb
aum
15h30-16h
Coffe
cor
ner
16h-17h30
Sym
posi
um (p
.41)
Mar
gare
t Pric
eRe
conc
eptu
alis
ing
Feed
back
in
Hig
her E
duca
tion.
Rutg
er K
appe
(p.43
)Ac
adem
ic a
sses
smen
t cen
ters
an
d st
uden
t com
pete
ncie
s: Re
liabi
lity,
cons
truc
t, co
nseq
uent
ial a
nd p
redi
ctiv
e va
lidity
.
Julie
Bro
snan
(p.45
)Ea
rly li
tera
cyas
sess
men
t and
Inte
rven
tion.
Mar
y H
ill (p
.47)
The
impa
ct o
f a d
edic
ated
as
sess
men
t cou
rse
on
deve
lopi
ng te
ache
r edu
catio
n st
uden
ts’ a
sses
smen
t cap
abili
ty.
Jam
es P
elle
grin
o (p
.49)
A Va
lidity
Fra
mew
ork
for
Asse
ssm
ents
Des
igne
d to
Sup
port
In
stru
ctio
n.
Vale
ntin
a G
rion
(p.44
)Sc
hool
qua
lity
eval
uatio
n:
stud
ents
’ poi
nts o
f vie
w
Mar
y Sh
eard
(p.46
)Eff
ects
of S
elf-
Pace
d Le
arni
ng o
n Ac
hiev
emen
t in
Sent
ence
Leve
l Wor
k in
Ye
ar 5
.
Katr
ien
Stru
yven
(p.48
)Th
e el
ectr
onic
por
tfolio
as a
n in
stru
men
t for
the
deve
lopm
ent
of te
ache
r com
pete
nces
in p
re-
serv
ice
teac
hing
pra
ctic
e.
Susa
n O
rr (p
.49)
Ded
ucin
g ‘le
arni
ng o
utco
mes
in
actio
n’ fr
om le
ctur
ers’
asse
ssm
ent
talk
.
Carm
en V
idal
Rod
eiro
(p.45
)D
o di
ffere
nt a
sses
smen
t ro
utes
(lin
ear v
s. m
odul
ar)
prep
are
stud
ents
, in
the
sam
e w
ay, f
or fu
rthe
r stu
dy?
Dav
id H
idal
go (p
.47)
Prac
tical
pro
blem
s in
the
asse
ssm
ent o
f tex
t co
mpr
ehen
sion
.
Lisa
Sm
ith
(p.48
)St
uden
t and
Teac
her E
duca
tor
Perc
eptio
ns o
f Bec
omin
g As
sess
men
t-Ca
pabl
e Te
ache
rs.
Fran
ces
Edw
ards
and
Bev
erle
y Co
oper
(p.50
)D
evel
opin
g ‘a
sses
smen
t cap
able
’ be
ginn
ing
seco
ndar
y te
ache
rs in
N
ew Z
eala
nd.
Thursday 30 august
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
5
Mai
n co
nfer
ence
room
Oly
mpi
a Bo
urse
Ba
llroo
m
Brea
kout
room
1
Euro
Brea
kout
room
1
Ster
ling
Brea
kout
room
1
Dol
lar
Brea
kout
room
1
Yen
9h-10h30
Invi
ted
Sym
posi
um (p
.27)
Shar
on K
linke
nber
gIn
vite
d Sy
mpo
sium
SIG
-DT,
Th
eme:
Test
-enh
ance
d le
arni
ng m
eets
lear
ning
an
alyt
ics.
Sym
posi
um (p
.28)
Lies
je C
oert
jens
Stud
ent a
ppro
ache
s to
lear
ning
and
thei
r pe
rfor
man
ce: E
vide
nce
from
prim
ary
and
high
er
educ
atio
n.
10h30-11h
Coffe
cor
ner
11h00-12h30
Sym
posi
um (p
.30)
Sue
Blox
ham
Asse
ssm
ent:
safe
in o
ur
hand
s?
Jan
Ard
ies
(p.33
)D
evel
opm
ent o
f an
Asse
ssm
ent T
echn
olog
ical
Li
tera
cy.
E. P
anad
ero
(p.35
)Fr
iend
ship
in th
e re
liabi
lity
of n
on-r
ubric
an
d ru
bric
-bas
ed p
eer
asse
ssm
ent.
Caro
lin R
amst
eck
and
Ann
ette
Fru
ehw
acht
(p.37
)A
Feed
back
Inte
rven
tion
Theo
ry
Pers
pect
ive
on E
xter
nal S
choo
l pe
rfor
man
ce D
ata
Use
in
Ger
man
Sec
onda
ry C
lass
room
s.
Sasc
ha W
üste
nber
g (p
.38)
Com
plex
Pro
blem
Sol
ving
in
Educ
atio
nal C
onte
xts –
Asse
ssm
ent,
Mea
sure
men
t Inv
aria
nce,
and
Co
nstr
uct V
alid
ity.
Carm
en T
omas
(p.33
)In
tegr
atio
n of
e-a
sses
smen
t pr
oces
ses f
or la
rge-
scal
e in
stitu
tiona
l dev
elop
men
t.
Robb
ert S
mit
(p.35
)Ru
bric
s as a
link
be
twee
n st
anda
rds a
nd
form
ativ
e as
sess
men
t.
Chan
g Zh
u (p
.37)
Prov
idin
g fo
rmat
ive
feed
back
to
stud
ents
via
em
ails
and
fe
edba
ck st
rate
gies
rela
ted
to
stud
ent m
etac
ogni
tion.
Gor
don
Joug
hin
(p.39
)M
ultip
le p
ersp
ectiv
es o
n a
viva
voc
e as
fina
l ass
essm
ent i
n ve
terin
ary
scie
nce.
Kris
Buy
se (p
.34)
Effec
ts o
f str
ong
port
folio
tr
igge
rs o
f heu
ristic
in
stru
men
ts a
nd li
nked
su
mm
ativ
e/fo
rmat
ive
asse
ssm
ent o
n w
ritin
g,
writ
ers a
nd tu
tors
.
And
ers
Jons
son
(p.36
)Sc
orin
g ru
bric
s for
st
uden
t lea
rnin
g.
Sara
h M
alon
e (p
.38)
Com
pute
r bas
ed a
sses
smen
t of
driv
ing
com
pete
nce
invo
lvin
g di
ffere
nt re
spon
se m
odes
Céci
lle R
aves
loot
(p.40
)Q
ualit
y im
prov
emen
t of s
umm
ativ
e ra
diol
ogic
al sk
ills a
sses
smen
t with
di
gita
l mul
ti-sl
ice
imag
es in
med
ical
ed
ucat
ion.
12h30-14h
Rest
aura
nt
Mid
twon
Gril
l
14h- 15h30
Keyn
ote
(p.40
)M
enuc
ha B
irenb
aum
15h30-16h
Coffe
cor
ner
16h-17h30
Sym
posi
um (p
.41)
Mar
gare
t Pric
eRe
conc
eptu
alis
ing
Feed
back
in
Hig
her E
duca
tion.
Rutg
er K
appe
(p.43
)Ac
adem
ic a
sses
smen
t cen
ters
an
d st
uden
t com
pete
ncie
s: Re
liabi
lity,
cons
truc
t, co
nseq
uent
ial a
nd p
redi
ctiv
e va
lidity
.
Julie
Bro
snan
(p.45
)Ea
rly li
tera
cyas
sess
men
t and
Inte
rven
tion.
Mar
y H
ill (p
.47)
The
impa
ct o
f a d
edic
ated
as
sess
men
t cou
rse
on
deve
lopi
ng te
ache
r edu
catio
n st
uden
ts’ a
sses
smen
t cap
abili
ty.
Jam
es P
elle
grin
o (p
.49)
A Va
lidity
Fra
mew
ork
for
Asse
ssm
ents
Des
igne
d to
Sup
port
In
stru
ctio
n.
Vale
ntin
a G
rion
(p.44
)Sc
hool
qua
lity
eval
uatio
n:
stud
ents
’ poi
nts o
f vie
w
Mar
y Sh
eard
(p.46
)Eff
ects
of S
elf-
Pace
d Le
arni
ng o
n Ac
hiev
emen
t in
Sent
ence
Leve
l Wor
k in
Ye
ar 5
.
Katr
ien
Stru
yven
(p.48
)Th
e el
ectr
onic
por
tfolio
as a
n in
stru
men
t for
the
deve
lopm
ent
of te
ache
r com
pete
nces
in p
re-
serv
ice
teac
hing
pra
ctic
e.
Susa
n O
rr (p
.49)
Ded
ucin
g ‘le
arni
ng o
utco
mes
in
actio
n’ fr
om le
ctur
ers’
asse
ssm
ent
talk
.
Carm
en V
idal
Rod
eiro
(p.45
)D
o di
ffere
nt a
sses
smen
t ro
utes
(lin
ear v
s. m
odul
ar)
prep
are
stud
ents
, in
the
sam
e w
ay, f
or fu
rthe
r stu
dy?
Dav
id H
idal
go (p
.47)
Prac
tical
pro
blem
s in
the
asse
ssm
ent o
f tex
t co
mpr
ehen
sion
.
Lisa
Sm
ith
(p.48
)St
uden
t and
Teac
her E
duca
tor
Perc
eptio
ns o
f Bec
omin
g As
sess
men
t-Ca
pabl
e Te
ache
rs.
Fran
ces
Edw
ards
and
Bev
erle
y Co
oper
(p.50
)D
evel
opin
g ‘a
sses
smen
t cap
able
’ be
ginn
ing
seco
ndar
y te
ache
rs in
N
ew Z
eala
nd.
Mai
n co
nfer
ence
room
Oly
mpi
a Bo
urse
Bal
lroo
mBr
eako
ut ro
om 1
Eu
roBr
eako
ut ro
om 1
St
erlin
gBr
eako
ut ro
om 1
D
olla
rBr
eako
ut ro
om 1
Ye
n9h-10h30
Invi
ted
Sym
posi
um (p
.51)
Liz
McD
owel
lAs
sess
men
t for
lear
ning
: va
ryin
g pe
rspe
ctiv
es w
ith a
fo
cus o
n le
arne
rs.
Sym
posi
um (p
.53)
Mar
ieke
van
der
Sch
aaf
Teac
her a
nd st
uden
t ro
les d
urin
g fe
edba
ck in
in
tera
ctio
n.
10h30-11h
Coffe
cor
ner
11h00-12h30
Gyo
ngyv
er M
olna
r (p.5
6)In
nova
tive
Asse
ssm
ent
Tech
nolo
gies
in E
duca
tiona
l G
ames
Des
igne
d fo
r In
tegr
atin
g As
sess
men
t int
o Te
achi
ng.
Gill
Elli
ot (p
.58)
How
mul
tiple
per
spec
tives
co
mpl
icat
e co
mpa
rabi
lity.
Jan-
Will
em S
trijb
os (p
.59)
Dev
elop
ing
a co
ding
sche
me
for a
naly
sing
pee
r fee
dbac
k m
essa
ges.
Dia
na B
aas
(p.61
)Pe
rcep
tions
of t
he p
urpo
ses
of p
ortfo
lio a
sses
smen
t in
prim
ary
educ
atio
n.
Ale
x Ko
zulin
(p.62
)D
ynam
ic co
gniti
ve a
sses
smen
t of
youn
g ad
ults
.
Lász
ló H
ülbe
r (p.5
6)Co
mpa
ring
diffe
rent
item
-ty
pes i
n pa
per-
base
d an
d co
mpu
teriz
ed e
nviro
nmen
t.
Ildik
ó Ba
lázs
i (p.5
8)Fi
rst r
esul
ts o
f Hun
gary
’s fu
ll co
hort
long
itudi
nally
co
nnec
ted
asse
ssm
ent.
Dan
iel G
arci
a-A
lmiñ
ana
(p.59
)Pr
ompt
Fee
dbac
k by
an
asse
ssm
ent i
nstr
umen
t (E
valc
omix
) in
an E
ngin
eerin
g D
egre
e
Mic
hiel
Vel
dhui
s (p
.61)
Prim
ary
scho
ol te
ache
rs’ u
se
of c
lass
room
ass
essm
ent
in D
utch
mat
hem
atic
s ed
ucat
ion.
Vict
oria
Que
sada
-Ser
ra (p
.62)
How
do
univ
ersi
ty te
ache
rs a
sses
s th
eir s
tude
nts?
Und
erst
and
asse
ssm
ent f
rom
thei
r ow
n vi
ewpo
int.
Gen
evie
ve G
auth
ier (
p.57)
Asse
ssm
ent P
lann
ing
for
Inte
ract
ive
Case
-bas
ed
Lear
ning
Sce
nario
.
Kris
tine
Høe
g Ka
rlse
n (p
.60)
Feed
back
mat
ters
whe
n st
uden
ts a
re co
nsid
erin
g w
hat t
ype
of le
arni
ng
activ
ities
to e
ngag
e in
- ex
plor
ing
the
rela
tions
hip
betw
een
feed
back
-typ
es a
nd
stud
ents
lear
ning
act
iviti
es in
hi
gher
edu
catio
n
Birg
it D
en O
uter
(p.63
)As
sess
men
t lite
racy
in u
nive
rsity
st
uden
ts: w
hat i
s it a
nd h
ow is
it
deve
lope
d?
12h30-14h
Rest
aura
nt
Mid
twon
Gril
l
14h- 16h
Deb
ate
Stan
dard
-bas
ed n
atio
nal
test
ing.
+ Cl
osin
g
16h-End
Coffe
cor
ner
Friday 31 august
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
6
Wednesday 29 augustKeynote
Up or Out and the Joy of Lifelong Learning. Joseph Kessels
Raising performance standards, evidence based instruction, compulsory professional develo-pment and strict accountability are recurrent topics in the debates on school improvement and the enhancement of educational quality. National and international rankings of student achieve-ments act as signposts for success and failure. Policy makers, school managers and heads of de-partments negotiate about financial means and learning outcomes. Assessment and evaluation play a crucial role in these debates. From a critical perspective assessment is probably the most political of all educational processes.
In view of the need for lifelong participation in learning and development it is relevant to explo-re the influence of external achievement pressures on students and their teachers. Is it a matter of up or out? Is an appreciative way of assessment helpful in establishing self-esteem, professional identity and a joy of lifelong learning?
Symposium
Symposium Chair: David Boud
There has been considerable interest on feedback in higher education worldwide prompted in part by consistent negative reports of surveys of student opinion that have been critical of uni-versities, and in part by a desire by scholars to make sense of a phenomenon that has been driven more by convention than by research. Useful review papers by Hattie and Timperley (2007) and by Shute (2008) have drawn attention to the existence of considerably more research on feedback than has been commonly recognised.
Conventional views of feedback are being found wanting. There is a need for new perspectives on feedback to be developed to recognise the diverse demands being placed on it. The sympo-sium papers argue for new and multi-faceted perspectives on feedback. The first provides a criti-que of the ways in which the idea of feedback has been appropriated from engineering without realising the full import of the idea and without taking account of the very different nature of students from engineering systems. It suggests that a new conception of feedback is needed for the educational context and sketches some of the key features of such a conception. The second paper also recognises the importance of the human dimension in feedback and argues that trust needs to be introduced as a key idea in how feedback is used and considered. Feedback is not only about the transmission of information from one person to another, but about the trustwor-
Symposium Discussant: Dai Hounsell
11h-12h30: Main conference room - Olympia Bourse Ballroom
11h-12h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
7
Paper 1.- Reframing feedback: new perspectives, new dimensions. David Boud
The paper argues that feedback needs to be reconceptualised to focus on desired educational outcomes beyond the immediate task. Feedback is a metaphor appropriated by education from its origins in other disciplines. This paper suggests that, in translation, this idea (Feedback Mark 1) has been misunderstood and misused in educational contexts, and that a vital feature from its engineering usage has been lost in its use in education. It argues that it is not a simple matter of reinstating the missing concept, useful though that might be. A more thorough reappraisal is needed in recognition of learners as human agents who cannot be regarded as inanimate systems susceptible to the feedback intentions of others.
The paper explores then, what a new conception of feedback, native to the field of education, could look like that was (Feedback Mark 2). It integrates recent scholarship of feedback and as-sessment that emphasises the importance of developing in students a sense of agency regarding their own learning, dialogue between givers and receivers of feedback in clarifying and apprecia-ting standards and criteria, educational sustainability of feedback so that it contributes to lear-ning beyond the immediate task, the use of multiple providers and receivers of feedback, and the design of educational programs to embed feedback activities utilising these features as a normal part of teaching and learning activities (from Boud, Carless, Hounsell, Nicol, etc.). It also incorpora-tes from Feedback Mark 1, the feature most commonly absent in education: the detection of the effect of interventions on subsequent production. In doing so it positions the effective use of fee-dback not just as a learning device, but as an intended learning outcome for the curriculum. The paper concludes by discussing the implications of such a reframing of feedback for educational practice and for research on assessment.
David CarlessTrust has long been recognized as an important issue in various fields, including economics,
political science, sociology and organizational theory. Trust is insufficiently discussed in relation to teaching, learning and assessment in higher education; and when it is mentioned it is often a minor sub-theme rather than a topic of extended discussion. The aim of this paper which is con-ceptual in nature is to analyze some key dimensions of trust in relation to feedback.
Many common barriers in the feedback process relate to ineffective communication or lack of trust. These include challenges to affect and the emotions; different perceptions of feedback amongst students and teachers; and some student misgivings about the role of peers in provi-ding feedback. For feedback to be effective lecturers and students need to enter into open and dialogic relationships of trust.
thiness of both provider and recipient of that information. Students cannot be expected to act on information if they do not believe that in doing so will be in their best interests. The third paper recognises the key role of student agency in feedback and links feedback with self-assessment undertaken by learners as a means for utilising information from teachers.
Paper 2.- Trust, distrust and their role in the feedback process.
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
8
The paper analyzes two dimensions of trust which carry relevance for the feedback process: communication trust and competence trust. Communication trust includes: empathetically lis-tening and responding sensitively to the thoughts of others; communicating frequently so that trust is developed through repeated interactions; and confidence that a feedback-provider has the integrity to behave fairly and ethically.
Competence trust or ‘trust of capability’ involves investing faith that others have the ability and the willingness to provide useful feedback on performance. Feedback from individuals with rele-vant expertise is more likely to be accepted and acted upon.
Two strategies are discussed which carry potential to develop trust: the further development of assessment literacy and enhanced communication processes. The development of assessment literacy amongst both teachers and students can strengthen competence trust. Enhanced trans-parency and communication about feedback may assist students in developing a deeper unders-tanding of the feedback process.
Paul OrsmondHigh achieving students in comparison to non-high achieving students are effective in using
tutor feedback in their learning due to their ability to self-assess. Therefore, encouraging students to learn from tutor feedback does not necessarily require a direct focus on the feedback provided. Rather it involves developing student’s ability at self-assessment. Tutors need to provide scaffol-ding for students so they recognise the importance of self-assessment in their learning. One way of doing this is to provide course infrastructure when designing curriculum, learning activities and assignments so that both tutors and students are able to focus on the role of ‘self’ in learning. A framework for achieving this is the GOALS process.
The GOALS process is student focused considering learning from a student perspective in the now: G = what outcomes do I need to Grasp. These can be self generated outcomes or tutor outco-mes which as translated by the student so that they understand them. O = how can I Orientate to ‘self’ perhaps through asking questions such ‘What am I trying to achieve?’ A = what Actions do I need to take to achieve my outcomes. This could be seen in terms of ‘who do I need to speak to’ or ‘how can I most effectively use the tutor provided exemplars’. The GOALS process also considers learning for the future. L = what type of Learning evaluation do I need to make. Asking questions such as ‘How have I done?’ or ‘How am I different now compared to when I started?’ ‘What has changed?’ Finally S = what Strategies do I need to develop in order to successfully move on. Stu-dents need to think about the experience of doing an assignment, perhaps in terms of how they can reduce the ‘error’ in their approach.
Paper 3.- Learning from tutor feedback: a role for self-assessment.
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
9
From an educational effectiveness perspective, research in the analyses of international compa-rative data has been limited as it neglects to take macroeconomic and developmental differences of countries into account. Further, methodological challenges of cross-sectional studies have as yet prevented the evaluation of international comparative data from a longitudinal effectiveness perspective. Therefore public league-table reporting of results yields biased judgments of suc-cessful and less successful educational systems.
In this paper methodological approaches commonly applied in Educational Effectiveness Research (EER) are used to comparatively analyze international large-scale achievement data. Drawing on data from the Progress in Reading Literacy Study (PIRLS) 2001 and 2006 it adjusts for non-malleable factors, creating the basis for “fair comparisons”. Further, it investigates how effectively educational systems grow, i.e. change, in their performance. Specifically, a methodolo-gical approach known from school effectiveness research is applied to investigate changes at the country-level with different cohorts of students.
Data from PIRLS trend countries is analyzed, applying a three-level hierarchical linear model that nests students in schools, in countries and includes a cohort variable at school level. Effective-ness measures of achievement status in 2006 and of changes from 2001 to 2006 are investigated.
Results suggest that there are countries which outperform their expected outcomes (status and change) as well as countries which stay below it, changing the picture of “high” and “low” perfor-ming educational systems, when contextual conditions and prior performances are taken into account.
The study enhances methodological developments of educational effectiveness research in cross-national assessments. Its results are relevant when international large-scale assessments are used to compare educational systems on the basis of their performance and yield a more reliable foundation for looking at policies, practices, structures, and ideologies that have helped countries to be successful, i.e. effective.
Paper A New Look at the Evaluation of Sociological Theories in International Large Scale Educational Assessments.
Daniel Caro • Andrés Sandoval-HernandezThis paper uses a mixed approach of theory generating and theory testing to evaluate theo-
retical models of economic, cultural and social capital. The data stem from two large scale in-ternational assessments: PIRLS 2006 and PISA 2009. In line with the mixed approach employed, the method is a combination of exploratory factor analysis (EFA) and structural equation models (SEM) known as exploratory structural equation modeling (ESEM). The main advantage of using this method is that an EFA measurement model with rotations can be used in a SEM, instead of a
Paper How effective are educational systems? A value-added approach to study trends in PIRLS.
Jenny Lenkeit14h-15h30: Breakout room 1 Euro
14h-15h30: Breakout room 1 Euro
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
10
PaperRevisioning National Assessment in New Zealand.
Alison Gilmore • Jeffrey Smith • Charles DarrThe National Monitoring Study of Student Achievement (NMSSA) is a new programme in New
Zealand designed to assess the school progress of primary school children at year 4 (roughly 8 years old) and year 8 (roughly 12 years old). It is a low-stakes, light-sampling approach that inclu-des both paper and pencil measures, performance-based tasks, interviews, and affective as well as cognitive assessments. The programme replaces the internationally recognised National Edu-cation Monitoring Project, which ran from 1995 to 2009. The programme is currently completing a developmental year, and will begin formal assessment activities in late 2012.
NMSSA is conducted jointly by the Educational Assessment Research Unit (EARU) of the Uni-versity of Otago, and the New Zealand Council for Educational Research (NZCER). The conceptual framework for NMSSA uses innovative approaches to national assessment that extend traditional approaches in terms of what is assessed, how it is assessed, and how it will be communicated. One of the goals of the new programme will be to move beyond simply assessing achievement to uncovering the underlying reasons for that achievement and for why it might differ in different segments of the population. The EARU/NZCER collaborative team has been working on the deve-lopment of this new programme for over a year as of the time of this proposal.
The purpose of the proposed presentation is to lay out the ideas that have been developed for the programme, with a special emphasis on new approaches to assessment, and new thinking on extending the notion of assessment to include why achievement occurs in addition to how much has occurred. We will discuss the theoretical framework of the programme in addition to presen-ting the results of an extensive series of pilot studies.
Paper Effects of Keyboarding Skills on Performance on Computer-Based L2 Writing Tests.
Khaled BarkaouiA major concern with computer-based (CB) tests of second-language (L2) writing is that perfor-
mance on such tests may be influenced by examinee keyboarding skills. From a cognitive pers-pective (e.g., Kellogg, 1996), poor keyboarding skills may force examinees to focus their attention and cognitive resources on motor activities (i.e., typing) and, consequently, other processes and
14h-15h30: Breakout room 1 Sterling
confirmatory factor analysis (CFA) measurement model. When using the EFA we have the advan-tage of being able to relax the zero loading restriction, allowing variables to load on different fac-tors simultaneously. The SEM then provides model fit indices, yields measurement invariance tests across countries, and corrects factor scores for measurement error. The results provide statistical criteria to evaluate the fit of theories with the empirical data (i.e., more reliably and validly). The paper contributes to the discussion on the advantages and disadvantages of using these methods when evaluating sociological theories in the context of international large scale assessments.
14h-15h30: Breakout room 1 Euro
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
11
aspects of writing (e.g., organization, revising) might be left unattended to, which can lead to poor text quality and lower test scores. Such effects might be more pronounced for L2 writers (Wolfe & Manalo, 2005). Few studies, however, have examined the impact of keyboarding skills on exami-nees’ performance on CB L2 writing tests.
This study investigated the impact of keyboarding skills on examinees’ test scores and writing processes. Each of 100 examinees (25 examinees x 2 L2 proficiency levels x 2 keyboarding skill levels) responded to two TOEFL-iBT writing tasks (independent and integrated) on the computer. A keystroke logging program recorded all writing sessions. Each essay was then rated by two in-dependent TOEFL-iBT raters.
Keystroke data were coded in terms of students’ pausing behaviors. Pauses provide observable and measureable cues as to where and how long writers halt in order to plan and revise their texts, whereas the linguistic output provides cues as to the nature of the hidden revision and planning processes (Matsuhashi, 1981). For example, long pauses are likely to be associated with macro-planning processes, whereas shorter pauses reflect microplanning processes (e.g., lexical choices). Pause were coded in terms of frequency, duration, type (formulation or revision pause), and tem-poral and linguistic location. Test scores and pause data were then statistically compared across keyboarding skill groups, L2 proficiency levels and task types. Findings and their implications for CB L2 writing assessment and research will be discussed.
Paper Developing national standards for the assessment of writing in com-pulsory education.
Astrid Birgitte Eggen • Synnøve MatreThe presentation reports from a project aiming at carrying out in-depth research on specified
assessment norms for writing, and their subsequent effects on the quality of pupils’ writing and teachers’ rating. The project builds on a social semiotic construct of writing that yields construct and content validity as well as principles for organizing standards and a multidimensional scoring instrument.
The project is designed in two stages. The objective of stage one is, in collaboration with 8 schools across Norway, to define which levels of writing competency should reasonably be ex-pected at specified grades as a basis for setting explicit standards. The objectives of stage two (intervention) are, in collaboration with 16 schools, a) to investigate what effects a shared set of explicit standards for educational writing, across nationally distributed local learning ecologies, may have on the quality of teacher assessment, in particular for rater agreement, and b) to inves-tigate what effects an integration of shared explicit standards in teaching may have on the quality of pupils’ writing. In stage two qualitative and quantitative methods are combined, and the study is designed as a repeated- measurements field study with six partially overlapping sub-studies.
Methods and findings will be presented from stage one of the project. Qualitative data were collected from a number of schools, using semi-structured small-group and individual interviews with teachers, supplemented by pupils’ texts. Findings, i.e. levels of writing competency expected at grades 5 and 8 of the compulsory school as to various dimensions of a written text (commu-nication, content, composition, use of language, encoding and presentation), will be presented.
14h-15h30: Breakout room 1 Sterling
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
12
Paper Examining Task Equivalence in L2 Writing Tests: A Mixed MethodsApproach.
Khaled BarkaouiExamining and establishing task equivalence in second-language (L2) writing tests that employ
various tasks across test occasions and/or offer examinees task choice is an essential step to en-sure valid score interpretations and fair decisions across examinees, tasks and testing occasions. This study illustrates how the question of task equivalence in L2 writing assessment could be em-pirically examined using both qualitative and quantitative data. We collected 46 essays on two ar-gumentative tasks, that are assumed to be equivalent in terms of their difficulty, but that differ in terms of orientation (private vs. public), from 27 Arab L2 university students. The essays were then rated by six of the students’ L2 writing teachers. Building on Connor and Mbaye’s (2002) Writing Competence Model, we developed a framework for analyzing the essays in terms of their gram-matical, discourse, sociolinguistic and strategic features. We then compared essay features and scores across writing tasks and examined the relationships between essay features and scores. The public task was more difficult and led to relatively higher inter-rater agreement, more diver-se and sophisticated vocabulary, more formal features, and better argument quality. The private task resulted in higher scores, longer essays and more informal features. The texts also differed in terms of their rhetorical structure across tasks. Finally, raters varied significantly in terms of the scores they assigned and their perceptions of the essays. The paper discusses the main findings of the study as well as the data collection and analyses strategies used in the study, particularly the operationalization of the Connor-Mbaye Writing Competence Model, and how they could be used in research on task equivalence and variability in examinee and rater performance in L2 wri-ting assessment.
Paper Spanish as Foreign Language Teachers’ conceptions ofassessment: preliminary results from an internet inquiry.
Gavin Brown • Ana RemesalWe present preliminary results of a study on the conceptions of assessment of ELE-Teachers
(ELE = Spanish as Foreign Language) from around the world. An online questionnaire was sent out via an online professional community with over 7500 members (Comunidad Todoele). The questionnaire got a response ratio of 57%; hence, the sample is 4275 respondents, representing different professional contexts: (1) teaching Spanish as Foreign Language in compulsory school, (2) as an extra-academic activity for children and teenagers; (3) in adult education, and (4) within in-company training.
The questionnaire of conceptions of assessment was designed following a four dimensional and bi-polar model (Remesal, 2006; Remesal, 2011). Conceptions are understood as the subjecti-
14h-15h30: Breakout room 1 Dollar
14h-15h30: Breakout room 1 Sterling
The results should initiate a discussion about 1) the relationship between integration of explicit standards of writing performance and the effects on pupils’ writing, and 2) their utility value as a basis for implementing nationwide tests of writing.
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
13
ve sum of individual beliefs, which, in turn, are assumptions about objects and phenomena that people take as true (often without intellectual contrast) (Green, 1971; Pajares, 1992). This model of conceptions differentiates among beliefs about the effects of assessment on teaching, on lear-ning, on certifying knowledge, and on account giving. For each of these dimensions, the beliefs about assessment can aggregate either as an overall conception of ‘assessment as an instrument for positive regulation of the teaching and learning process’ or a conception that ‘assessment is an instrument for societal control’.
Exploratory factor analysis of the data point to significant differences between the four teaching contexts: in-school, extra-school, adult education, and in-company. It also reveals differences in conceptions of assessment in relation to the teachers’ teaching practice (traditional top-down lan-guage teaching versus communicative bottom-up model), and their own experience as assessed subjects during their own school experience. Thus, the results highlight a need for a specific edu-cation for assessment that helps the teachers develop assessment competence if assessment for learning is the goal.
Paper Assessment in Brazilian higher education: Validation of a Portuguese version of the Students’ Conceptions of Assessment (SCoA-VI)inventory.
Gavin Brown • Daniel A. S. Matos • Walter L. Leite • Sergio D. CirinoStudents’ conceptions of assessment have been investigated with a self-report survey inven-
tory that aggregates 33 items into 8 factors forming four major conceptions (i.e., assessment is for improvement, assessment brings social/personal benefit, assessment is irrelevant, and as-sessment is a reflection of external attributes). A Portuguese version of the Students’ Conceptions of Assessment (SCoA) inventory was administered in two Brazilian universities (N=756). Confirma-tory factor analysis used the mean and variance-adjusted weighted least squares estimator and good fit to the original model, developed in New Zealand with secondary students, was found. Students also indicated the practices they associated with assessment; four types of assessment practices were found: informal student-controlled; informal teacher-controlled; formal student-controlled; and formal teacher-controlled. The students emphasised conventional, formal evalua-tion methods controlled by the teacher. Even supposedly student-controlled practices (i.e., self and peer-assessments) were perceived as teacher-controlled practices. As a whole, the university students agreed most with assessment being for improvement, and second for accountability. The definitions of assessment were meaningfully predicted by key factors in the SCoA inventory in a well-fitting structural equation model. Together these relations suggest that students think: “Formal assessments are not enjoyable, but they are not bad because they help me improve my learning. If assessment is informal and controlled by the teacher, it’s good for class dynamics and climate of the classroom, and I do not ignore this type of assessment activity.” The level of educa-tion, language, and country did not interfere with how the SCoA items bundled together to make latent traits indicating that the same latent constructs exist in two samples from quite different populations. The structural pathways between SCoA factors and definitions of assessment are un-
14h-15h30: Breakout room 1 Dollar
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
14
Paper Conceptions of Classroom Assessment: Findings from UsingQ-Methodology.
Wei Shin LeongThis paper details the pilot use of Q-methodology (Q) for understanding the different possible
conceptions of classroom assessment among Singaporean teachers. This work builds on an earlier study involving the phenomenographic method of interviewing teachers on their perspectives of assessment. To my knowledge, Q has not been used in education assessment research even though it is known to be a useful methodology for studying different teachers’ perspectives. My interest is currently in the tacit decision-making process in wide ranging social interplays among teachers in the practical situations that Q researchers tend to prefer to investigate. The pilot the-refore serves at least two main broad purposes: firstly, to challenge myself to try new methodo-logy in my current stage of the research journey into teachers’ classroom assessment and invite feedback from experts en route. Secondly, to study the process and results from the use of Q and what I have learnt that could have possible implications for my future research work and for other researchers, teachers and policy-makers. A key question is whether the Q is helpful to investiga-te the possible multiplicity and significant relationships of conceptions of classroom assessment within a specific context of Singaporean teachers’ complex classroom decision-making environ-ment, whereby teachers are empowered to make significant decisions on curriculum, pedago-gy and assessment. The different clusters of conceptions from the Q-factor analysis, in particular, have revealed underlying bases of common and different views of classroom assessment and implications for practices. This suggests opportunities for consensus-building that will be useful for teachers, policy-makers, professional learning developers and researchers.
Paper Teacher Communication of Evaluation Results: Impact on Student Engagement in School.
Carolina Carvalho • Joseph Conboy • Maria Odete ValenteStudent engagement is a factor in students’ school enjoyment and success; it may influence de-
cisions to stay in school and provide an opportunity for at-risk students to achieve social equality. However, some evaluation practices may act to reduce student engagement. Secondary educa-tion in Portugal is marked by high-stakes testing that can influence acceptance in higher edu-cation programs. In this study, we listened to what some student voices say about evaluation practices in use in secondary schools in Portugal. Tenth- and 11th- year students (N=105) in two secondary schools in the Lisbon area, provided structured reflective narratives about the impact
derstood to be ecologically rational. Students express belief patterns that are consistent with the ways assessment is used in their educational contexts.
The authors acknowledge financial support to the first author from Coordenação de Aper-feiçoamento de Pessoal de Nível Superior (CAPES).
14h-15h30: Breakout room 1 Dollar
14h-15h30: Breakout room 1 Yen
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
15
(both as students and persons) of types of communication teachers use when reporting evalua-tion information. An initial reading of the student responses indicates that students have few po-sitive episodes to relate about teacher assessment. Students clearly object to unjust interpreta-tion of their academic effort and condemn abusive and humiliating actions on the part of some teachers, as well as teacher sarcasm and irony when presenting evaluation results. According to the students, a poor grade can increase engagement if it is accompanied by teacher support and belief in the student’s ability; it can lead to disaffection if the low grade is unexpected, perceived as unjust, or does not correspond to the effort exerted. Students indicate that they prefer clear information from teachers that points the way as to how to improve. They point out that a sum-mative classification does not provide this information. Poor human relations on the part of some teachers may also lead to student disaffection.
PaperThinking about Fairness in Classroom Assessment.
Robin TierneyFairness is an enduring ideal and a perpetual concern in educational assessment. Until recently,
most research relating to fairness focused on standardized tests or large-scale assessments. Grea-ter interest in classroom assessment has raised questions about fairness, particularly because of the personal nature of some methods, the interaction involved in the use of assessment for learning, and the need to serve diverse students. While classroom assessment was once conside-red low-stakes in comparison to standardized testing, it is now widely recognized that teachers’ judgements can have profound consequences for students. Fairness issues are often mentioned in classroom assessment literature, but clear definitions of the concept are less common. Inter-pretations of fairness vary considerably, and they are often reported by group membership (i.e., teachers, students), which suggests that fairness is a matter of perspective. However, by linking different perspectives, fairness may be more productively considered as a single concept with multiple, interrelated facets. This research brings the perspectives of highly experienced and spe-cialized teachers into a scholarly dialogue, and draws on their practical wisdom to reconceptualize fairness for the dynamics of classroom assessment. Practical wisdom (phronesis) is essentially a fluid network that uses different forms of knowledge, including theoretical, practical and ethical knowledge, to determine the best course of action in particular circumstances. The six participants in this multi-case study were purposefully selected based on years of teaching experience, type of degree, and level of specialization. While each offered a unique perspective on fairness, patterns were evident across their recommendations. Six aspects of fairness, involving transparency, stu-dents’ opportunities, equitable treatment, critical reflection, and the classroom environment, are discussed in relation to a model of assessment for learning. The goal in presenting this work is to facilitate thinking and generate discussion within the educational community about the meaning of fairness in classroom assessment.
14h-15h30: Breakout room 1 Yen
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
16
Roundtable Accuracy in Self-assessment: Debating the Merit of Accuracy when Students Evaluate their Own Work.
Gavin Brown • Heidi L. AndradeThis roundtable will present divergent views on the importance of accuracy in student self-eva-
luation. The dominant view in Assessment for Learning policies and interpretations of classroom assessment is a pedagogical interpretation of assessment arguing that student self-assessment of work products and processes is useful for raising academic performance. This view draws on self-regulation of learning theories of how students improve their own work against socially-defined goals. Students can learn to set targets, evaluate progress relative to target criteria, and conse-quently improve the quality of their learning outcomes. Hence, there has been a strong adoption of student self-assessment in classroom practices, with little or no concern for the validity of stu-dents’ self-assessments. .
PaperThe emotional dimension of feedback.
Carol EvansThis study focuses on the emotional dimension of feedback in order to better understand how
to enhance feedback practice within higher education institutions (HEIs) and specifically addres-ses the question of how to enhance student engagement in and with feedback. An interpretative phenomenological analysis approach was adopted (Smith, 2003) whereby the subjective expe-riences of student teachers (STs), to inform programme development, was the focus. The two year study involved 34 postgraduate STs in two UK HEIs. The findings of the first year study were used to inform development of the second year feedback intervention as part of an action research project funded by the Higher Education Academy, UK. Data gathered via group and individual interviews, as well as email communications were analysed using content-analysis procedures to identify over-arching themes (Krippendorff, 2004). The emotional dimension of feedback was one of the emergent meta themes, identified as underpinning how students accessed and made use of feedback opportunities. Implications for policy and practice from the study are twofold. Firstly, a number of elements in the design of assessment and feedback were identified that could be enhanced to reduce student anxiety, particularly in relation to feed-forward holistic assessment design elements. Secondly, in developing sustainable feedback practice, the importance of stu-dent self-regulation of feedback is essential (Boud, 2000; Hounsell, 2007) and management of the affective dimension of feedback is imperative in this respect. It was evident that “savvy fee-dback seekers” were more able to manage the emotional dimension of feedback compared to their peers. In developing assessment practice as an integral element of teaching, it is important that HEIs address the emotional dimension of feedback explicitly with students in order to sup-port them in developing strategies to maximise their ability to self-regulate their learning in order to engage in and with feedback.
14h-15h30: Breakout room 1 Yen
16h-17h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
17
Introduction of quality assurance elements into the assessmentpolicy of generic graduate attributes of university using the course-embedded assessment methods.
However, the reliability of testing is considered a sine qua non for the validity of assessment interpretations, and research into psychological processes underlying the human ability to self-evaluate work raises serious doubts about the quality of students’ judgments. It has been shown that novices (i.e., students) tend to lack sufficient knowledge by which to evaluate their own work appropriately and, when possessing that knowledge, they are likely not to make use of it. Addi-tionally, psychological safety factors indicate that self-assessment can be compromised by inter-personal relations present in classroom environments. Furthermore, since progress in many edu-cational domains is relatively ill-defined, it is difficult for learners, let alone instructors, to validly evaluate progress or status.
While self-assessment is important, there is still a debate as to how accurate such evaluations need to be. While research has shown how accurate student self-assessment can be, little has been done to investigate whether it is necessary that the self-assessments are accurate. There are different perspectives on how to move forward with implementing self-assessment in classrooms. This roundtable will examine quality issues in this important classroom assessment practice.
Roundtable
Patrick LaiDeveloping graduate generic competencies is one of the major responsibilities of universities.
Literature on generic graduate attributes focuses on studies of Barrie in (1) understanding of ge-neric attributes by university teachers; (2) revision of university’s statement of generic graduate attributes and (3) development of a conceptual framework for the teaching and learning of ge-neric graduate attributes. There is NO study that discusses valid and reliable methods to assess generic graduate attributes on an institutional basis.The significance and originality of this pa-per extends literature on generic graduate attributes by exploring and evaluating the usefulness of course-embedded assessment (CEA) as an institutional learning outcomes measure, with two possible directions of using CEA for institutional outcomes assessment. On-going observations and interviews with 11 university teachers from 7 departments throughout the project and a final round of post-project interviews with 9 key players of the project team for data triangulation were done. Interviews prompts for the post-project interviews focus on (1) recommendations for using course-embedded assessment (CEA) as a tool for institutional assessment; (2) strategies to enable valid, reliable data collection and (3) pros and cons of the two suggested possible directions of using CEA for institutional outcomes assessment were included.Findings of the observations and post-meeting interviews address the issues of (1) whether it is possible to use programme selec-ted artifacts to assess outcomes at institutional level and the changes that need to be made to en-sure consistency; (2) the extent to have common criteria and standards with respect to outcomes across programmes and a preferred way to make the process reliable and valid. Finally two pos-sible directions with respect to an institutional framework of outcomes assessment is discussed.
16h-17h30: Main conference room - Olympia Bourse Ballroom
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
18
Roundtable The relationship between past and present in end of term tests in Flemish history education.
Karel Van Nieuwenhuyse • Kaat Wils •Geraldine Clarebout • Lieven VerschaffelEmpirical research on the way in which a school subject is assessed, gives a privileged entrance
into teaching practices. It enables to go beyond official curricular goals and teachers’ statements about the subject, and gives insight into what teachers consider in the end to be important goals to achieve and competences. Our presentation will be based on the analysis of a corpus of 200 written end of term tests in history for the 11th and 12th grade in schools within the Flemish Com-munity (the Dutch-speaking, northern part of Belgium). As such, it connects to the first strand of the conference, on summative assessment.
The analysis will focus on the position of the present and the relationship between past and pre-sent in the end of term tests. Teachers in Flanders are explicitly encouraged to integrate current issues in their lessons and to ‘actualize’ their teaching on the past (which, according to the official curricula, in the 11th and 12th grade, deals with the period from 1789 to the present). At the same time, however, they are expected to stimulate students to think historically, and hence to pay at-
RoundtableWhat Types of Assessment Contexts Support Self-Regulated Learning?
Huiyong Seah-Tay • K.H. Koh • K. TanThe propose paper examines the relationship between self-regulated learning (SRL) and
authentic assessments using a mixed method design, with sequentially quantitative and qualita-tive phases, to answer the research questions: 1) Was there a difference in students’ use of SRL in a conventional paper-and-pen task and an authentic assessment task? 2) How was SRL facilitated or hindered by real life setting, school setting, student control and the student’s ability to self-assess? To cross-validate the findings, the study was replicated two years later with another group of par-ticipants at the same site. Quantitative data on SRL was gathered using a self-report questionnaire I had designed based on Zimmerman’s (2002) cyclical Forethought, Performance and Self-reflec-tion SRL model. Participants, comprising 405 secondary four students in the first study and 300 in the second, were assigned two tasks that tapped on the same cognitive skills, except that one was a paper-and-pen task while the other was an authentic assessment task. Comparisons of the SRL scores between the two tasks, using paired sample t-tests, showed a small statistically significant difference in favour of the authentic assessment task and none in the second. Further analysis at the subscale and phase level added a more nuanced understanding of the relationship between the two phenomena. These inferences were supported by interview data from 19 participants: 6 from the first study and 13 from the second. Their comments corroborated the literature review that real life setting, student control and ability to self-assess facilitated students’ SRL. Furthermo-re, the study provided a clearer picture of how these factors are linked. These findings suggest reframing our view on formative and summative classroom assessments, especially with regards to authentic assessments.
16h-17h30: Main conference room - Olympia Bourse Ballroom
16h-17h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
19
Roundtable Understanding of Assessment by University Lecturers: how this links to theories.
Maddalena Taras • Mark DaviesThis paper reports work in progress on a series of studies that examines lecturers’ understan-
dings of aspects of assessment in a UK HEI. These aspects include summative and formative as-sessment processes, products and functions and the relationship between them, and how fee-dback and learner involvement relate to them.
Empirical data were collected from 50 education lecturers, 50 science lecturers and 12 academic developers through questionnaires and also through semi-structured interviews. The data were examined qualitatively and quantitatively to provide an overview of understandings of aspects of assessment, feedback and learner involvement.
This paper focuses on the differences and similarities in understandings between the education and science lecturers in order to ascertain if greater access by education lecturers to pedagogic theories increases the understandings of different aspects of assessment.
Due to the inconsistencies in the education literature, the research posits that education lec-turers will reflect the representations in the literature and therefore have an incomplete and un-harmonious understanding of assessment terms and how they relate and interrelate. As concerns science lecturers, it is posited that without the conflicting theories that education lecturers are required to engage with, it will result in a high degree of coherence in their reasoning and unders-tandings.
Initial evaluation of the results indicates that there is a difference in understanding between education and science, although the differences are not as great as were originally predicted. By exploring and evaluating the results, this paper provides an overview of the thinking and unders-tanding of lecturers and examines how this will impact on practice and therefore students.
It further permits the participants to also question their own understandings and positions on some of these issues and therefore consider how they can refine their assessment procedures and the support of students’ learning.
tention to the specific historical context of past phenomena. This tension holds a major challenge for teachers. Previous research among (future) history teachers suggests moreover that they are more geared towards the present in their general beliefs on history education than in their actual teaching practices.
The in-depth analysis of the end of term tests will shed light on the extent to which teachers pay attention to the present in their concrete educational practice, and how they approach the pre-sent and its relation to the past. Based on this analysis, intervention strategies will be proposed to achieve a (more) conscious and balanced handling of this tension, to enhance students’ learning of history.
16h-17h30: Main conference room - Olympia Bourse Ballroom
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
20
RoundtableFormative Assessment using Collective Portfolios.
Núria de SalvadorIn the context of competence-oriented learning, virtual portfolios using wikis can be appropria-
te tools to carry out formative assessment. However, motivating pupils to produce in a wiki increases the assessment load of the teacher
and decreases his or her control over pupils’ productions. The leitmotiv of this field research is to reflect on means to facilitate assessment in this context,
and offer technical and methodological solutions to it. This proposal arises from a real need for a course in teaching English in secondary and VE education, where wikis are used as virtual portfo-lios for the last two school years.
Roundtable High School Student’s Portfolios in Science for All: Indicators of the Quality of their Teachers’ Work.
Amira Rom • Michal NachshonThe Curriculum for Science for All is based on processes which make connections between tea-
ching, learning and evaluation. Its goal is to instill science technology literacy into high school students who do not specialize in science. The main tool used in the evaluation process is that of the portfolio which includes representative products of the students’ work, as evidence of the learning process, and the students’ achievements and progress. The products are a reflection of their scientific literacy and include complex individual and group assignments which are evalua-ted by means of a competency rubric.
The teacher’s role in school is to find alternative means of problem solving for students, in ac-cordance with his more pronounced area of intelligence. Everything points to the need for a wide ranging change in outlook on learning achievement evaluation, encompassing changes in tea-ching strategies, the teachers’ role and in the teacher-student relationship.
Evaluation of the teachers’ work is an important component of the educational process. When attention is devoted to the evaluation process and its influence on teaching and learning, a practi-ce evolves which positively effects and improves the quality of these process. .There are two main purposes in evaluating a teacher’s work: improved performance and establishment of accounta-bility.
The study goal was to investigate students’ portfolios as indicators of the quality of their tea-chers work.
The findings point out the strong and weak points of the teachers and suggest guide lines for future teacher training and preparation in order to best implement alternative evaluation in the field.
As a result of these findings the research suggests ways to improve the quality of work of the teachers and the use of the portfolio as a system wide evaluation tool in Science for All classrooms specifically and in secondary schools in general.
16h-17h30: Main conference room - Olympia Bourse Ballroom
16h-17h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
21
Masahiro Arimoto • Janet Looney
Roundtable Exploring undergraduate students’ informal, formative peerassessment networks.
Rita HeadingtonPeer assessment incorporates peer-marking, peer-correction, peer-rating, peer-feedback, peer-
review and peer-appraisal (Topping 1998). Updating Topping’s study, Gielen et al’s (2011) inven-tory demonstrates increased application and diversity of peer assessment practice integral to un-dergraduate students’ academic studies. Gielen et al (2011) do not discuss students’ transferral of taught peer assessment methods or skills to informal contexts, yet it is known (eg HEFCE, 2007; Ryan 2008, Sutton and Taylor 2011) that students use online and face-to-face social networks, of peers, family members and work colleagues, to undertake informal, formative peer assessment of coursework and assignments.
Nicol (2010) sees peer assessment as supplementary to tutor-student interaction, not separate from it. Additionally, Orsmond and Merry (2009) identify differences in high and low achieving stu-
This paper presents case studies of small rural schools in Akita, one of Japan’s top-scoring pre-fectures in the 2007 National Assessment of Academic Ability. The authors focus on the concept of “formative assessment” for lesson study. A framework on formative assessment to support “lear-ning to learn” is extended to include Wenger’s concept of the “communities of practice”, The paper describes mutual engagement, joint enterprise and the development of shared repertoires within these communities.
The results of the case studies showed that (1) teachers use jargon based on the Japanese “ki-zuki” to refer to embedded assessment, rather than more formal assessment criteria or methodo-logy. They also try to incorporate the Japanese concept of “neriai” (polish, elaborate, work out or knead lesson well) into the core pedagogical culture. “Neriai” means a discussion phase with the whole class discussion phase for structured problem solving. It is the core of teaching through problem solving. It is accompanied with teachers’ double loop learning and support for learning circles based on social capital. (2) a “soft middle” level allows for modifications to take place in policy making; teacher-initiated approaches combines elements from different subjects, so that “bottom-up” approaches undergo a transformation throughout the process of implementation. “Learning to learn” it is discussed more extensively. The municipalities have built system leader-ship around this concept to open the door to other experts and expand frontiers (e.g. removing the barriers to welfare). They are formally accountable to the national bureaucracy, and also an intimate and very real accountability to their colleagues. Teachers are involved in their students’ lives outside of school and are in constant communication with the parents, they are accountable to the parents and grandparents in a unique way.
Linking a framework of formative assessment and evaluation for “learning to learn” to learning circles for transformational school leadership.
Roundtable
16h-17h30: Main conference room - Olympia Bourse Ballroom
16h-17h30: Main conference room - Olympia Bourse Ballroom
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
22
Roundtable Students’ underperformance due to pseudo understanding of math-ematical concepts: What can be improved in school assessment?
Ho Yi XianPseudo-understanding describes a phenomenon where students show evidence in apparent
understanding of particular concepts, but have yet achieve thorough grasp of the topic or sub-ject matter. While pseudo-understanding allows students to perform well at summative tests and exams, it poses a critical problem when they progress to higher and deeper levels of learning. Gaps in their prior understanding of topics, concepts and ideas hinder students’ learning and in-ternalizing of new knowledge. A qualitative study of four pre-university students, who had scored distinctions in their previous national examinations but did not do well in their college tests and exams, revealed that one of the key obstacles for the construction of new knowledge was stu-dents’ pseudo-understanding of mathematical concepts from their lower levels of study. Despite students giving written answers and working which could be considered acceptable and correct, clinical interviews with the students revealed gaps and misunderstandings in several concepts which they had previously learnt in secondary school. This may be a result of assessment prac-tices during their secondary school years. It was found that students were unable to self-assess, they were not habitual in reviewing their work, and they do not question, explain, defend, reason and challenge their understanding of mathematical concepts. Assessment practices which may have contributed to students developing pseudo-understanding include excessive repetition of tasks which resemble those given in summative tests and rushing for students to attempt exam typed questions in multiple timed pen and paper testing, even before students could make sense and abstract ideas from manipulations. Shift of focus to formative assessment and assessment for learning can encourage students to reflect on their learning and challenge conceptions that they formulated which minimizes pseudo-understanding of mathematical concepts and facilitate knowledge construction at higher levels of learning.
dents’ commensurate abilities to interpret and make use of feedback. As these studies highlight tutors’ roles in managing group composition to facilitate effective peer assessment within social constructivist frameworks, is informal, student-led peer assessment inevitably a deficit model of practice?
Set in a UK university, and forming the first part of a longitudinal study of students’ interpreta-tion and use of formative and summative feedback, this study explores undergraduate initial tea-cher education students’ informal, formative peer assessment networks, focusing on ‘composition of assessment groups’ and ‘interaction between peers’ (Gielen et al 2011). Using social network analysis (Knoke and Yang 2008; Carrington and Scott 2011), it investigates who students choose as informal peer assessors within their personal networks, how the formative assessment usually takes place (eg online, face-to-face) and what students hope to gain by choosing to work with particular individuals.
This paper provides outcomes, identifies and discusses issues emerging from the study and cri-tiques the methodology. It considers the appropriateness of students’ informal networks for peer assessment and the support they require to use networks effectively.
16h-17h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
23
Roundtable Will digitally aiding the creation of a project plan createtransfer?
Steven Nijhuis
Roundtable UITZIGT integrated assessment of competences on outflow level in teacher education.
Kristien Carnel • Katelijne De Munter • Ilse Loockx • Ann Adriaenssen • Van Heddegem Ilse • Bieke Quaghebeur
In competence learning we need assessments in which knowledge, skills and attitudes are inte-grated. In competence oriented education we see a growing need of a more integrated approach to final assessment, if possible in authentic learning contexts.
UITZIGT (Uitstroomcompetenties voor zij-instromers geïntegreerd toetsen) is funded by the School of Education (SoE) within the K.U.Leuven association 2010-2012. The project aims to de-velop a toolkit for the qualitative assessment of competences at the end of the training program based on teaching internships.
Based on a concise literature research we investigated which assessment tools are best suited for assessing certain basic competencies and competence clusters referring to the professional profile of a teacher. This lead us to the development of a blueprint UITZIGT, an assessment battery to test competences of oncoming teachers as they are ready to finish.
Teacher education is increasingly confronted with students that apply for short track programs. APEL procedures and more competence based curriculum designs increase the need for proper
To support students in higher education in creating a project plan for their project based lear-ning assignments, a Digital Coach was created and successfully implemented. The Digital Coach promotes reflection of the students in preparing the project plan.
Project plan benefitted greatly from the students use of the coach, as could be expected [1, 2]. In this workshop the context of the coach (project based learning, design research, use) is descri-bed and the coach is used.
Focal point of the discussion is whether the Digital Coach supports learning, specifically the importance of creating a sound project plan. It cannot be concluded that the coach supports transfer. Which gives rise to the question how to research the effect of transfer and even whether this research will prove to be useful [3].
[1] Paans, W. (2011). Accuracy of nursing diagnoses: Knowledge, knowledge sources and reaso-ning skills. (PHD, Katholieke Universiteit Leuven).
[2] Paans, W., Sermeus, W., Nieweg, R., & van der Schans, C. (2010). Determinants of the accuracy of nursing diagnoses: Influence of ready knowledge, knowledge sources, disposition toward criti-cal thinking, and reasoning skills. Journal of Professional Nursing, 26(4), 232-241.
[3] Eiriksdottir, E., & Catrambone, R. (2011). Procedural instructions, principles, and examples: How to structure instructions for procedural tasks to enhance performance, learning, and transfer. Human Factors, 53(6), 749-770.
16h-17h30: Main conference room - Olympia Bourse Ballroom
16h-17h30: Main conference room - Olympia Bourse Ballroom
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
24
RoundtableAssessing procedural Knowledge of German Business Students.
Christine Caroline JähnigAssessment of student competencies is a highly relevant topic for research, practitioners and
policy makers. Unfortunately the number of available competence tests for German universities is rather limited. In the field of Business Administration Studies only a small number of standar-dized tests is available that primarily focus on assessing declarative knowledge. Considering the competence orientation of the Bologna Process and the complex demands young business pro-fessionals face when leaving the university, a need for action-oriented complex assessment is postulated. This contribution introduces an innovative test type to assess procedural knowledge of undergraduate business students. Procedural knowledge is the knowledge exercised in the accomplishment of a task, and thus closely related to the action-oriented competencies. Unlike declarative knowledge, it cannot be easily articulated by the student and is often described as tacit. The test combines elements of Situational Judgement Tests (SJTs) with approaches of tacit knowledge measurement. The challenge of meeting demands of curricular and criterion validity when constructing a situational test for higher education will be discussed.
instruments for final assessment that better suits alternative learning pathways typical for these target groups.
Funded by a project of School of Education of the Catholic University of Leuven 3 institutes for Teacher Education (KHLeuven, CVOVIVO Kortrijk, Lessius Mechelen) developed an assessment ba-ttery that responds to these intentions.
The UITZIGT project aims to assess the outflow competences based on alternative assessment procedures. Based on a concise literature research we investigated which assessment tools are best suited for assessing basic competencies and competence clusters referring to the profes-sional profile of a teacher. This lead us to the development of a blueprint UITZIGT, an assessment battery to test competences of oncoming teachers as they are ready to finish their study program.
This paper provides outcomes, identifies and discusses issues emerging from the study and cri-tiques the methodology. It considers the appropriateness of students’ informal networks for peer assessment and the support they require to use networks effectively.
RoundtableAuthenticity and portfolio assessment.
Michael Bots • Jolanda de PutterWithin Dutch higher education many teacher education programs rely on the use of electronic
portfolios for assessing professional development and competence. Despite the growing research in this area, many practitioners and researchers are still looking for the best way to combine stan-dardization for optimal reliability with an open structure that encourages authenticity.
This paper attempts to contribute to the discussion by presenting the results and implications
16h-17h30: Main conference room - Olympia Bourse Ballroom
16h-17h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
25
Godfrey Pell • Richard Fuller • (Matt Homer)Institutions have conflicting pressures of quality, feasibility and cost in the assessment of stu-
dents. Students desire a demonstrably fair assessment process which is, ideally, completed by all in a single academic year. This most typically takes the format of assessment of the whole student cohort, followed by a period of remediation and retesting for those who underperform.
However, resit/borderline candidates are not a homogenous group and considerable difficulty persists in accurately identifying and profiling these ‘at risk’ students. Emergent work also reveals that traditional models of test-remediate-retest may not lead to sustained long term improve-ment in performance for these students, raising major implications in assessment policy for edu-cational institutions (Pell et al 2009; Hauer et al 2009).
This interactive workshop will overview existing literature in this area, and encourage parti-cipants to recognise the limitations of their own assessment strategies and methodologies for underperforming students vis-a-vis competent students, and the implications for policy. The de-velopment of sequential test methodologies, based on regression towards the mean phenome-na (Bland & Altman, 1994) will explore a way of meeting this challenge. Participants will explore theoretical and practical applications of sequential testing methodology, and gain confidence in practical approaches to managing cost-benefit analyses and quality issues in assessment policy.
A strong focus of the workshop will be the discussion of the limitations of existing research in this field, and encouragement of participants to share their own good practice and opportunities for collaborative research in this important field. This workshop has particular significance for all disciplines using performance based/criterion based assessment approaches.
of a research study in the Netherlands. Using a combination of qualitative research methods the project has investigated the current use of portfolio assessment in the field of teacher training and the perceptions of students, assessors and supervisors on (the authenticity of) different portfolio structures and procedures. The results provide a deeper understanding in the way students reflect and anticipate on the assessment procedure, and how assessors arrive at a conclusion by using all kinds of (and more or less relevant) information.
In the round table session we will present the results and discuss some of the preliminary con-clusions on guidelines that will improve the authenticity of portfolio assessment. Together with the audience we will discuss the implications of these guidelines for the reliability, efficiency and transparency of portfolio assessment.
The challenge of meeting the assessment needs of both competent and underperforming students within existing educational structures: Introducing sequential testing? Lessons for policy and practice.
Workshop 1
16h-17h30: Breakout room 1 Euro
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
26
Workshop 3 A theory of improvement for teachers’ professional development in Assessment for Learning.
N. van Benthum • J. Van der Steen • J. Gulikers • F. De Jong • M. MulderVocational education in The Netherlands is engaged in the introduction of competence-based
education (CBE) (De Bruijn & Leeman, 2011). In this reform, it’s needed to focus on Assessment for Learning (AfL) which first priority is to serve the purpose of promoting students’ learning (William, 2011). However, teachers are usually not trained in designing AfL in CBE (Popam, 2001; Birenbaum, 2003). Therefore, it’s crucial to distinguish features which stimulate teachers’ professional develo-
Workshop 2EvalCOMIX: A web service for u-feedback and u-feedforward.
Eduardo García • Mª Soledad Ibarra • Gregorio RodríguezThe EVALfor research group has developed a series of projects based upon the principles of
learning oriented assessment (Carless, 2006) and sustainable assessment (Boud, 2000). One of the main products of this research is the web service EvalCOMIX (http://evalcomix.uca.es), specifically designed and developed to aid teachers and students in implementing participative assessment activities. EvalCOMIX achieves this using strategies such as self-assessment, peer- assessment and co-assessment.
This workshop provides the necessary information and resources for the participants to use the EvalCOMIX web service. Once completed, participants will have acquired the following skills:
•Design of assessment instruments (checklists, rating scales, semantic differentials and rubrics).•Design assessment instruments that promote feedback and feedforward.•Use of instruments for teacher assessment processes, self-assessment, peer- assessment and
co-assessment.•Analyse results from the assessment tasks undertaken.
The workshop has been designed as an essentially practical exercise in which participants de-sign and develop their own assessment instruments for use in Learning Management System (LMS). To this end, participants will initially be presented with the basic characteristics of the Eval-COMIX web service (design, management, import and export of assessment instruments). The first part will then conclude with a practical exercise in which each participant will develop at least one rating scale and one simple rubric.
After that, a presentation will be given on ways to use the designed assessment instruments, in order to perform teacher assessment, self-assessment, peer- assessment and co-assessment.
Finally, participants will be presented with the possibilities EvalCOMIX offers for analysis of re-sults obtained from an assessment process, by means of analysis of the different graphs presented by EvalCOMIX.
All the instruments offered by EvalCOMIX can be used not only on a personal computer, but also on other more current devices such as smartphones or tablets. EvalCOMIX thus transcends the li-mits of e-assessment and brings forward the beginning of ubiquitous assessment (u-assessment).
16h-17h30: Breakout room 1 Sterling
16h-17h30: Breakout room 1 Dollar
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
27
pment in AfL in a theory of improvement (ToI). A ToI is a description of how features of a pro-fessionalization program stimulate teachers’ learning and is often absent in teacher professional development (Van Veen, Zwart, Meirink en Verloop, 2010).
In this study, a ToI is designed to professionalize teachers in AfL. The ToI is based on a literatures-tudy and is applied in a program to improve teachers’ and managers’ assessment practice in Dutch Agricultural Vocation Education and Training (AVET).
Characteristics of the ToI are: 1. Conceptions on assessment which influence how teachers deal with assessment in practice (Ho, Watkins & Kelly, 2001; Segers & Tillema, 2011). 2. Assessment practice because classroom assessment practice is the change context (Smith, 2011) and should be challenged by giving feedback on actual practice (Gardner et al, 2010). Conceptions are cha-llenged in teachers’ assessment practice and puts the teachers in a constant 3. inquiry to impro-ve practice (Birenbaum et al, 2011). 4. Congruent professional development in which teachers learn to apply AfL in classrooms and managers to apply AfL to stimulate teachers’ learning in as-sessment. This characteristics of the ToI lead to the most suitable approach for this professional development in a Professional Learning Community (PLC). A PLC is closely related to the concept of collaborative learning communities.
Study is running and preliminary results show potential of this ToI to improve assessment prac-tice to AfL.
Thursday 30 august
Research on the development of expertise performance (Ericsson, 2006, pp. 683–703) has shown that people improve their performance considerably if they regularly do specific exerci-ses that are adjusted to their ability level and include immediate feedback. In this digital age and using computer adaptive practice (Klinkenberg, 2011) it is possible to monitor the ability of stu-dents while they are in their learning process. To deliver immediate feedback it is necessary to know what kind of errors an individual makes. This calls for analysing error types, indicating the relative frequency of these errors and provide feedback. In this invited symposium we will explore where test-enhanced learning meets learning analytics.
Sharon Klinkenberg • Marthe Straatemeier • Han van der MaasIn this paper we present a model for computerized adaptive practice and monitoring. This mo-
InvitedSymposium
Invited Symposium SIG-DT, Theme: Test-enhanced learning meets learning analytics.
Symposium Chair: Sharon Klinkenberg Symposium Discussant: Silvester Draaijer9h-10h30: Main conference room - Olympia Bourse Ballroom
Paper 1.- Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation.
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
28
Silvester DraaijerThe need to increase student success rates, especially in the first years of higher education, is
becoming stronger and stronger. Online quizzing in the first years of undergraduate education is regarded as one possible way to improve time-on-task, student engagement and retrieval ability. Several studies are published that address the incorporation of online quizzing in higher edu-cation. These studies show mixed results with regard to study success. For example, in general, student opinions towards the use of online quizzes is positive, however the effect on time-on-task is just modest and the correlations with success rate vary from no correlation to significant corre-lations with modest effects. The studies incorporate various course set-ups and qualitative and quantitative analyses methods and take a different number of variables into account. These could be the cause of the different findings. In this paper, an overview of several studies is presented and discussed. Online quizzing will be related to the ideas of deliberate practice and possibilities of data-mining and feedback (learning analytics) to improve the possible generation of data on which students and educators can act on and to increase the impact of online quizzing.
Symposium Student approaches to learning and their performance: Evidence from primary and higher education.
Due to its clear educational relevance, the link between student approaches to learning and their study outcomes has been frequently investigated in the context of higher education (e.g., Vermunt, 2005; Ward & Walker, 2008). Yet, the results on this issue do not speak in one voice: mixed results have been reported. The manner in which academic performance in operationalized could play a key role in explaining these mixed findings. By using different measures for academic per-formance, the first study contributes significantly in verifying this hypothesis.
The relationship between student approaches to learning and performance has been often looked at during the period of higher education. The question arises however whether students’ approaches have already predictive power earlier on in the educational curriculum. Focussing on the learning conceptions on students in primary education, the second study asks the whether
Symposium Chair: Katrien Struyven Symposium Discussant: Liesje Coertjens
del is used in the Maths Garden, a web-based monitoring system, which includes a challenging web environment for children to practice arithmetic. Using a new item response model based on the Elo (1978) rating system and an explicit scoring rule, estimates of the ability of persons and the difficulty of items are updated with every answered item, allowing for on the fly item calibra-tion. In the scoring rule both accuracy and response time are accounted for. Items are sampled with a mean success probability of .75, making the tasks challenging yet not too difficult. In a pe-riod of ten months our sample of 3648 children completed over 3.5 million arithmetic problems. The children completed about 33% of these problems outside school hours. Results show better measurement precision, high validity and reliability, high pupil satisfaction, and many interesting options for monitoring progress, diagnosing errors and analyzing development.
Paper 2.- Using online quizzing and quiz results to improve study success.
9h-10h30: Breakout room 1 Euro
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
29
Mikaël De Clercq • Benoit Galand • Mariane FrenayThe aim of this study was to investigate the impact of motivational and cognitive processes on
several achievement outcomes. Academic achievement was assessed through final percentage and three tasks assessing three different levels of knowledge (acquisition of knowledge, unders-tanding of principles and application of concepts). The relation between achievement and self-efficacy beliefs, goal orientation, learning strategies and self-regulation was investigated while controlling for student’s previous performance. The participants were 217 students from the en-gineering faculty at the Université catholique de Louvain in Belgium. The students were assessed at the beginning of their third year of study. Data were analyzed through stepwise regression. The results mainly highlighted that final percentage is essentially modulated by motivational fac-tors (mastery goal), whereas the performance to specific tasks was related with cognitive factors (deep processing and surface processing). Moreover, the results also revealed that the impact of self-efficacy beliefs on achievement outcomes vanished when student’s previous performance are controlled for. The implications of our results on the understanding of the relation between learning process and academic achievement have been discussed.
Eric Robbers • Peter van Petegem • Vincent Donche • Sven De MaeyerGiven the fundamental character of Learning Conceptions and the fact that learning concep-
tions develop overtime into conceptions that are found in secondary education pupils and in higher education students (Marton, Dall’Alba & Beaty, 1993), insight into learning conceptions of young pupils is relevant especially in correlation with learning results.
The objective of this research is to determine whether there is correlation between learning conceptions of pupils in grade eight of primary education and their learning results. The sample contained 649 twelve years old pupils in grade eight of primary education spread among 16 pri-mary schools in the south east of the Netherlands. Learning conceptions were measured with the revalidated learning conceptions questionnaire (authors). Learning results were measured using the primary school education final progression test. Given the hierarchic structure in the data multilevel analysis was used.
Research findings show that children in grade eight of primary education who scored high on the scales of negative attribution and external regulation of the learning conception questionnai-
the predictive validity holds at this age as well. This would offer possibilities for early remedial activities.
The link between of student approaches and performance is not only of substantive but also of methodological interest. In longitudinal studies on the change in learning strategies, this me-chanism generates a biased follow-up sample. Such selective sample could in turn jeopardize the correctness of the estimates of change in learning strategies over time. The third study investi-gates whether the missingness can be considered not completely at random, using the data of a three-year study in a Flemish University College.
Paper 1.- Learning processes in higher education: Comparing the impact of cognitive and motivational variables on specific and global measures of achievement.
Paper 2- Correlation between learning conceptions and final school results in primary education.
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
30
Liesje Coertjens • Vincent Donche • Sven De Maeyer • Peter Van PetegemLongitudinal studies on the change in learning strategies during higher education are on a rise
(Vanthournout et al., 2011). As is common with this design, there is an amount of attrition (Sha-dish, Cook, & Campbell, 2002). By applying listwise or pairwise deletion prior to analysis of change over time (e.g., Zeegers, 2001), most longitudinal studies take the assumption that this missing-ness is completely at random (MCAR) (Enders, 2010). The students who did not participate in all waves are thus considered a random subset of those persisting in higher education and in the research. Yet, given the literature on the link between learning strategies and learning outcomes achievement (e.g., Furnham, Monsen, & Ahmetoglu, 2009; Yip, 2009), this assumption is questio-nable. Recent methodological advances allow for estimating whether missingness can be seen as not at random (MNAR) or non-ignorable (Enders, 2011). This study innovates in applying this to a longitudinal study on the growth in students’ approaches to learning during higher education. One cohort of Flemish University College was followed during their course in higher education and were administered the ILS-SV (Donche & Van Petegem, 2008) at each of the three years.
Preliminary results indicate that for the relating and structuring, self-regulation, external regu-lation scales the MCAR assumption does not hold. The missingness can be assumed dependent on the previously observed scores for these scale (e.g. at the first wave). Moreover, for the lack of regulation, self-regulation and external regulation the missingness is estimated being not at ran-dom. Implications of these results are discussed.
Symposium Assessment: safe in our hands?
Higher education assessment is a massive, global, edifice involving students, academics and administrators as millions of assignments, exams, performances and other tasks are assessed and graded annually. At the heart of this unstoppable behemoth are institutions’ and subject discipli-nes’ academic standards. Confidence in these standards has been questioned in recent years in se-veral in several HE systems (Brown, 2010; Coates, 2010, Vanderslice, 2004) prompted by a growing diversity and complexity in higher education (Woolf & Cooper, 1999). There are claims of grade inflation and lack of parity of standards as Universities and academics largely retain the right to set and maintain their own standards. So, is assessment safe in our hands?
Symposium Chair: Sue Bloxham Symposium Discussant: Clair Hughes
Paper 3- Is missingness in longitudinal research during higher education dependent on student’s learning strategies? A Flemish case study.
re, scored significant lower on the final school test for primary education. For children in the last year of primary education, learning results are significant lower when they have a high score on the variables negative attribution and regulation by teachers. The fact that learning conceptions change after transition from primary to secondary education (Klatter, 2003), means that the sti-mulation of pupils learning conceptions in the direction of the correct form of regulation, using more profound processing activities possibly leads to better learning results.
11h-12h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
31
Sue BloxhamThis paper considers academic standards in assessment; an imperative at a time of political and
reputational pressure in the standards arena. Assumptions underpinning standards are discussed drawing on contrasting theoretical perspectives; the techno-rational perspective which under-pins quality assurance of assessment and the hermeneutic account which emphasises the inter-pretive nature of academic judgement.
Neither epistemological approach provides a satisfactory basis for grading policy which is illus-trated by the contrast between the assumptions underpinning quality assurance of standards (re-liable, consistent, transparent) and the everyday practice of grading (shifting, socially constructed and tacitly held standards). Yet millions of assessments are graded unproblematically each year. The paper will consider how these contrasting perspectives are negotiated by lecturers in the act of grading work as a basis for developing a policy agenda for academic standards in use.
It draws on a mixed-method study of faculty from four disciplines and 3 universities. ‘Think aloud’ protocols and interviews were utilised to access both actual and espoused data regarding grading practices. The findings illuminate how lecturers are implicitly combining different epis-temologies in negotiating work practices. Whilst recognising that much of what they do is inter-nalised and tacit, assessors gain a sense of security about their marking judgements from various quality assurance documents and procedures. The act of creating and discussing, for example, assessment criteria provides a clear opportunity for detailed dialogue about standards. There is a strong influence of ‘techno-rational’ artefacts on interpretive practice as many quality assurance processes for assessment become normative.
However, the findings continue to reinforce the local and co-constructed nature of standards albeit within a framework of dialogue about explicit statements of standards . The implications for developing appropriate policy regarding assessment standards, consensus, accountability and information for students will be discussed.
This symposium brings together three papers which investigate this question from theoretical, policy and practice perspectives. The first paper examines the failure of existing theory to provi-de a workable model for standards in use. It explores the lack of alignment between empirical evidence regarding academic judgement processes and the assessment information provided to students. The second paper provides an insight into approaches to policy implementation regar-ding standards. It explores features of a range of nationally-funded projects designed to enhance the understanding and use of standards in Australia’s evolving regulatory environment. The final paper explores professional judgement at the practice level. It reports on an empirical study re-garding how different members of the academic profession come to know academic standards as operationalised in their grading of student work.
The symposium debate will draw on commonalities in the papers regarding ways forward for policy in rebuilding confidence in assessment, for example using communities, dialogue and exemplars. It will provide the opportunity to debate how standards might be more effectively understood and used in the academy at different levels and with different audiences.
Paper 1.- “You can see the quality in front of your eyes”: developing a policy agenda for academic standards in use.
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
32
Margaret PriceThe problems of shifting from norm-referenced assessment to criterion-referenced assessment
have been identified by several scholars in recent years. These important critiques touch on a number of areas, but neglect a key question about how assessors learn to accomplish what Shay calls a ‘socially situated interpretive act’ (2004). Criterion-referenced assessment is premised on clear standards applied consistently by all assessors in a community. However the limitations in making standards explicit and the consequent problems in sharing common understanding of standards means that the criterion-referenced assessment remains ‘faulty’ especially where the-re are multiple assessors working together. New assessors or incomers to a local community of markers have to quickly try to assimilate the standards in use in order to belong. Research which does exist tends to focus on salaried, full-time academics. This overlooks the heterogeneity of the academic labour force in HE, and the substantial contribution made by contract (hourly-paid) lec-turers, particularly in applied disciplines such as Business and Management.
This study explores how nine newcomers to a UK Business School - including salaried and con-tract staff - attempt to understand local assessment practice. We use a situated learning lens to analyse their diary entries and interviews about their experiences of learning to mark. Drawing
Clair HughesAn intensification of interest in the quality of teaching and learning in higher education has led
to international trends in defining and articulating the standards of academic achievement (Harris 2009). While commonalities can be observed among the ways the standards agenda has been given expression in many parts of the world- for example the development of quality frameworks and the emergence or refocusing of regulatory bodies –distinctly individual national priorities, activities and support structures have also emerged.
An investigation of standards-related projects supported by the (recently abolished) Austra-lian Learning and Teaching Council (ALTC) was undertaken as the situational analysis stage of the Assessing and Assuring Graduate Learning Outcomes (AAGLO) project (Barrie, Hughes and Crisp 2011). Though the issue of ‘standards’ had been a central or peripheral concern of many ALTC pro-jects since its 2004 inception (Oliver 2011), the Learning and Teaching Academic Standards (LTAS) project (ALTC 2011a) was the first ‘standards’ project coordinated on a national scale.
This paper briefly introduces the LTAS project and Australia’s evolving higher education regu-latory arrangements (TEQSA 2011) in order to contextualise an analysis of the large and diverse range of activity the ALTC has subsequently encouraged. The analysis of 11 ALTC grant projects and four fellowships that are currently addressing academic standards focuses on factors such as discipline, scope, sponsoring or leading body, the nature of outcomes and approaches to the conduct of the project and to the assurance of standards. The analysis makes sense of the multiple approaches and perspectives being delivered through this range of activity and adds a comple-mentary perspective to the standards discussions in the Australian higher education context.
Paper 2.- Current Australian approaches to assessing and assuring academic standards.
Paper 3.- Learning to mark: exemplars, dialogue and participation.
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
33
PaperDevelopment of an Assessment Technological Literacy.
Jan Ardies • Bruno Bossaerts • Sven De MaeyerAn increasing importance of broadly educated youngsters is emerging in Europe. Several coun-
tries (e.g. UK, The Netherlands, Belgium,...) are redesigning their national or regional compulsory curricula. Thinking about the need of science and technology in the standard curricula is a dis-cussion started in the seventies or even before in most countries (Fensham, 2004). Although the growing interest in the field of technology education is relatively young and little research has been done (Garmire & Pearson, 2006). Technology as a school subject is now in most countries mandatory in secondary and/or primary education. Nevertheless there is little research done about educational effectiveness in technology classes and the state of technological literacy of pupils. This research is aiming to develop a test to examine the dimensions ‘understanding’ and ‘making sense’ (Van Woensel, 2010) of pupils in order to make it possible to measure student outcomes concerning technological literacy. Several phases are undertaken to develop the test. Focus groups and individual interviews with experts in the field of technology education deter-mined a starting point and provided a set of questions. These questions were analyzed, mainly through the Item Response Theory based on a pilot study (n=251). A second round of analyzing the data based on new content analysis and a new sample of 2000 students resulted in a final test providing information about the status of technological literacy of this representative group. In this study we will report on analyses on the fit of the test-data with suitable IRT models and on the selection of proper items from a pilot test using a combination both the content-based and IRT-perspective. Finally we will report on a DIF-analysis, answering the question whether certain items function different for boys and girls.
11h-12h30: Breakout room 1 Euro
Paper Integration of e-assessment processes for large-scale institutional development.
Carmen Tomas • Michaela Borg • Jane McNeilTen years ago, e-assessment was predicted to become all-pervasive in higher education insti-
tutions. A combination of cultural, political, resource, and infrastructural constraints has resulted in a much slower take-up than expected. The current investigation aims to address the issue of
on scholars such as Sadler, we examine the suggestion that exemplars are important for newco-mers coming to understand local assessment practice. We argue for the fundamental importance of dialogue about exemplars and other aspects of practice – both to develop inter-subject un-derstandings of assessment ‘anchor points’, and also to foster among newcomers (and especially contract lecturers) a great sense of participating in, and being valued by, the local assessment community.
Shay, S. (2004). The assessment of complex performance: a socially situated interpretive act. Harvard Educational Review, 74(3), 307-329.
11h-12h30: Breakout room 1 Euro
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
34
Paper Effects of strong portfolio triggers of heuristic instruments and linked summative/formative assessment on writing, writers and tutors.
Kris BuyseIn this paper we describe two instruments in the assessment of writing, i.e. (1) strong portfo-
lio triggers for using heuristic instruments such as spelling checkers during the writing process and (2) a strongly linked summative/formative assessment, assessing the (summative) exams in exactly the same way as the (formative) take-home tasks. After outlining the context of writing as-sessment that gave rise to the experiments with the two instruments, we will present a first range of possible effects on the writing process and products of higher education students in Flanders, as well as on the writers and the tutors, i.e. how these assessment procedures affect and enhance student learning, writing assessment and organizational change.
The results of a survey as well as those of the analysis of the scores’ evolution suggest that the implementation of the first instrument is motivating and effective for writing activities and cour-ses among the students: it not only improves their writing products, but also enhances better and motivating writing and learning processes in general. Furthermore, it also affects the writing as-sessment of tutors, and indirectly organizational change, as they asked (and obtained) a different implementation of the writing course (in terms of conditions with regard to frequency, audito-rium and evalution system), based on the positive results of their methodology .
With regard to the second instrument, the same quantitative and qualitative analysis will be
e-assessment take-up from an assessment life-cycle perspective, with view to establishing a set of priorities for institutional development.
Twenty-six academic members of staff were interviewed on their uses of technology to support assessment, in order to develop an institutional perspective and to identify the barriers that pre-vented more larger scale use of e-assessment technologies.
The interviews revealed extensive and invisible uses of technology in development of the task (using email, internet etc) and communication stages (via VLE) of the process. The use of technolo-gy to communicate assessment to students was largely the result of a previous institutional policy.
The assessment event was found to vary according to different applications. Use of e-portfolios was in the main limited to certain disciplines and reflected limitations similar to its paper-based counterpart. Another e-assessment application, computer based assessment (CBA) encountered both infrastructural limitations and a perception that its suitability is at lower levels of study. These combined constraints relegated CBA to formative and low-stakes assessment.
Electronic submission of coursework was reported to be subsidiary to paper processes mainly for cultural reasons and practices. Whilst the technology means were available, cultural barriers around marking prevented the establishment of full electronic submission, marking and fee-dback. The other large scale application, exams, was limited by the infrastructure.
Given these barriers, stages in marking, entering feedback, and storage of assessment records and archiving were primarily paper based. The assessment life-cycle perspective on e-assessment facilitated insights into which areas are most problematic and which applications could most ra-pidly bring more gains to an institution.
11h-12h30: Breakout room 1 Euro
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
35
PaperRubrics as a link between standards and formative assessment.
Robbert SmitThe aim of the presented study is to examine the learning benefit of rubrics as part of the pro-
cess of formative assessment. We see rubrics as a beneficial tool to align lessons with standards (Burke, 2006). The rubric used in this project is embedded within the topic of “reasoning in mathe-matics” and in line with the Swiss standards of mathematics (Linneweber-Lammerskitten & Wälti, 2005).
Items in large-scale tests don’t allow for just a few open-ended questions with extensive written answers, as they need a sufficient number of items in order to reach satisfying reliability values. Nevertheless important mathematical competencies like problem solving or reasoning require time to search and explain. So it lies with the teacher to test these competencies in more authentic
Paper Friendship in the reliability of non-rubric and rubric-based peerassessment.
Ernesto Panadero • Margarida Romero • Jan-Willem StrijbosThere is an increase in the use of peer assessment in collaborative learning activities. One of the
challenges for the peer assessment acceptance among learners and teachers is the reliability of this procedure. In this study we explore the use of a rubric to enhance the reliability of the peer assessment. The rationale is based on the criteria and scoring features included on the rubric that could contribute to better assess a peer work. The social aspects of peer assessment are also stu-died by analysing the effects of the friendship between the evaluator and the assessed peer. The study involved 216 pre-service teachers from the Universitat Autònoma de Barcelona in their third year. A quasi experimental design was used where half of the students evaluated a peer concept map using a rubric while the other half evaluated without rubric. The results show a generalized tendency to over-score the peer works. Nevertheless, an increase of reliability in the rubric group was found when compared to the students’ assessing without the rubric, with the latter scoring significantly higher their peers’ work using the score given by the expert/teacher as the reference. To analyze the impact of friendship on the accuracy of the score students informed about their friendship degree towards the student they were assessing. Three friendship degrees were consi-dered (low, medium and high). For low and medium friendship degrees the students using a ru-bric-based assessment made a more accurate assessment than the non-rubric group. For the high level of friendship no differences were observed between the rubric and non-rubric students. We discuss the interest of the rubric-based assessment in the context of peer assessment with a spe-cial interest on the effects of this methodology for students showing low and medium degrees of friendship.
performed on the data of the current academic year 2011-2012, so that we are not able yet to an-swer the question, but at first glance the data of the first semester look promising.
11h-12h30: Breakout room 1 Sterling
11h-12h30: Breakout room 1 Sterling
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
36
PaperScoring rubrics for student learning.
Anders Jonsson • Ernesto PanaderoScoring rubrics seem to have the potential of promoting student learning and the gains in stu-
dent performance when using rubrics are sometimes conspicuously large. Still, besides the fact that there is a quite limited number of studies on the formative use of rubrics, there are also seve-ral limitations to this research. Starting from these limitations, this paper reviews current research on formative uses of rubrics. By reviewing such research both in greater detail and with a primary attention to influences on student learning, the aim of this paper is to investigate in what ways rubrics may support student learning and also to identify factors that may influence – either po-sitively or negatively – the effects of using rubrics to support student learning. Findings suggest that the use of rubrics may support student learning by: (1) aiding teachers in improving instruc-tion, (2) aiding the students in determining teacher’s expectations, (3) aiding the students in de-veloping self-regulated learning, (4) facilitating the feedback process, (5) lowering student anxiety about assignments, and (6) improving students’ self-efficacy. Findings also suggest that while the studies performed in higher education contexts report on positive results when providing the students with rubrics, the positive results in schools are linked to longer and larger interventions. One factor that seems to be of paramount importance, for school students, is therefore the time devoted to work with the rubric. There is also a apparent difference between studies reporting on small or partial effects of using rubrics on the one hand, which to a large extent use rubrics only for self-assessment, and studies reporting on larger gains, where a broader repertoire of instructional interventions are used. When it comes to rubric design, not many factors have been systematically investigated. However, findings suggest that rubric design may not be crucial for student learning.
and appropriate situations. Standard based rubrics can help to make criteria known to students and to assess mathematical reasoning.
Since our research project had the character of a pilot study we limited ourselves to a small sample of four primary classes. On the one hand we used quantitative methods to measure the students competency in mathematical reasoning. On the other hand we interviewed the teachers and groups of students to gain information about the success of our lessons with rubrics as a sup-porting learning tool.
Using Rasch-modeling we were able to validate the four levels of our rubric, but the reliability of the students’ competency level measures is rather low due to the few open items. The qualitative content-analysis of the interviews shows that all students have generated a basic understanding of the competency. They gained the insight that reasoning is not just wrong or right, but it can be done on different competency levels. The students have learned to self-assess themselves and the teachers felt supported by the rubric when giving feedback.
11h-12h30: Breakout room 1 Sterling
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
37
Paper A Feedback Intervention Theory Perspective on External School per-formance Data Use in German Secondary Classrooms.
Carolin Ramsteck • Annette FruehwachtDespite growing research literature on the effects of mandatory testing and performance fee-
dback systems, there is still a lack of theoretical understanding on how performance feedback works to improve instruction and student learning. This paper aims at exploring feedback inter-vention theory (FIT) as conceptual framework for modeling instrumental feedback data use in German secondary classrooms. Four FIT-related assumptions on effective external performance feedback data use were applied to qualitative teacher interview data. Data were gathered in 20 German grammar schools. The material was analyzed with qualitative content analysis methods (deductive category application in combination with inductive subcategory building). Results show that only some teachers are able to assign performance feedback data to specific aspects of their teaching (as focal task). Furthermore, feedback data does not operate as corrective feedback since teachers are not able to test fine-grained didactical hypotheses with feedback reports at hand. The paper concludes that FIT might be a valuable perspective to further investigate the flaws and potentials of current mandatory testing policies in Germany and other European coun-tries.
Paper Providing formative feedback to students via emails and feedback strategies related to student metacognition.
Chang ZhuAssessment is a central element in the overall quality of teaching and learning in higher edu-
cation. Formative assessment aims to enhance learning through providing appropriate feedback. Effective formative assessment helps the student to see the gap between their current understan-ding of course components and the goals of the course so that they can take the appropriate ac-tion to achieve these goals. Formative feedback can be provided to students through a variety of ways. Among all those methods, electronic feedback methods are increasingly used by teachers since they enhance feedback production, delivery and communication (Race, 2001). Electronic feedback has the advantage of providing timely and individualised feedback. Among the factors related to student learning, metacognition is a very important one as it makes the learner aware of his/her cognition and trigger control processes that serve the pursued goal of the self-regulation process (Efklides, 2008).
This research focuses on studying students’ preferences of formative feedback strategies and the effects of teacher’s formative feedback on student learning. Participants in this study were third-year undergraduate students (n=109) studying Business Management in a university in Bei-jing, China. The findings of this study indicate that providing the right kind of feedback to stu-dents can make a significant difference in their achievement. In addition, student characteristics such as metacognition level need to be considered in the adoption of different feedback strate-
11h-12h30: Breakout room 1Dollar
11h-12h30: Breakout room 1 Dollar
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
38
Paper Computer based assessment of driving competence involving different response modes.
Sarah Malone • Sarah Katharina Freidinger • Roland BrünkenOne empirical indicator of test validity is a clear difference between expert and novice test per-
formance. Some general findings of expertise research may help to develop such valid tasks. One of these findings is that experts outperform novices more clearly with increasing domain specifi-city of the given task–it’s similarity to the characteristic demands of a defined domain–(Glaser & Chi, 1988). This aspect implies that in order to make a statement on a participant’s skill in a cer-tain domain, the individual should be given a task that resembles the tasks that naturally arise in that domain. Some domains like e.g. sports and driving require fast decision making under time constraints (Hodges, Huys, & Starkes, 2007). For the assessment of expertise in such domains test items including an adequate response format seem to be crucial for the ecological validity of the test. The aim of the present study was to investigate whether the validity of the assessment of expert performance in car driving can be enhanced by an ecological valid response mode of the applied tasks. 35 learner drivers (novices) and 31 individuals with some driving experience (ex-perts) were to solve 21 items including identical animated driving scenarios. The response format for each scenario was varied experimentally: one half of the participants were to react to breaking hints within the animation (high ecological validity) and the other half worked on multiple-choice questions following the presentation (low ecological validity). We postulated that driving experts would outperform driving novices in both response modes. Moreover, the hypothesis that follows from the concept of domain specificity is that the mentioned expert-novice difference should be more obvious for the reaction time response format. Whereas the ANOVA revealed a main effect of expertise, no interaction between the factors expertise and response mode could be found.
gies. Feedback via email communications was considered a very convenient and timely manner for the teacher and students in this study. The current study finds that students with high and low metacognition reported a different level of preferences for feedback strategies; nevertheless most students prefer constructive and elaborative comments from teachers. Furthermore, the results show that teacher’s formative feedback was appreciated by students, and promoted student lear-ning.
Paper Complex Problem Solving in Educational Contexts –Assessment, Measurement Invariance, and Construct Validity.
S. Wüstenberg • S. Greiff • G. Molnár • A. Fischer • J.Funke • B. CsapóComplex Problem Solving (CPS) is seen as a cross-curricular competency, which has recently at-
tracted interest in large-scale-assessments. In the PISA 2012 cycle, CPS will be assessed across the world partly using minimally complex computer simulations based on the MicroDYN approach.
11h-12h30: Breakout room 1 Dollar
11h-12h30: Breakout room 1 Yen
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
39
In this talk, we present empirical results on MicroDYN, a computer-based test containing multiple items aimed to measure two main complex problem solving processes - knowledge acquisition and knowledge application. More specifically, we (a) test a measurement model for MicroDYN composed of the two aforementioned processes, we (b) evaluate whether measurement invarian-ce holds across different grade levels of high school students in order to compare latent means between grade levels and (c) investigate relations between CPS, reasoning, grade point average (GPA), and parental education. Analyses are based on N=855 Hungarian high school students in grades 5 to 11. Using structural equation models, results show that (a) a 2-dimensional model with the facets knowledge acquisition and knowledge application fit the data best. Furthermore, (b) CPS is strongly factorial invariant and thus, mean differences between grades can be interpreted meaningfully. Latent means in both facets increase at higher grades, disregarding a considera-ble drop in CPS performance in grade 9. Finally, (c) results based on path analyses indicate that knowledge acquisition incrementally predicts variance in parental education and – to a lesser extent – in GPA even beyond reasoning. A proper assessment of CPS in addition to subject related abilities is of high educational relevance. Results of this study provide important implications on how to use CPS in an assessment context.
Paper Multiple perspectives on a viva voce as final assessment in veterinary science.
Gordon Joughin • Michael McGowanThe viva has a long history. It has been, and remains a prominent form of assessment in many
European countries and is used across a wide range of disciplines in the UK, Australia and el-sewhere. The final year veterinary science program at an Australian university utilises a viva as the final piece of assessment in a five-year program. Students are assessed on their ability to com-petently investigate clinical case scenarios and communicate practical, economically responsible and scientifically valid recommendations on diagnosis, clinical management and control of a di-sease problem. Multiple perspectives on the viva were sought through the following process: (i) A random sample of students were interviewed immediately following their viva, with questions focussing on their expectations of what would happen, what they believed would be tested, how they prepared for the viva, how they would compare the viva with written examinations, and whether they would recommend it as a form of assessment. (ii) The four academic staff who con-ducted the viva in pairs were also interviewed about their perceptions of the viva, its purpose, advantages and disadvantages. (iii) The Faculty’s Associate Dean (Academic) was interviewed re-garding his perceptions of the viva. The three sets of perspectives of student, academic, academic administrator are presented in this study. Each set of perspectives serves to illuminate our unders-tanding of the viva as an engaging and revealing form of assessment. Comparing and contrasting the perspectives reinforced the value of the viva as a ‘rite of passage’ from the perspectives of all stakeholders, with students also drawing attention to the impact of the viva on their approaches to studying.
11h-12h30: Breakout room 1 Yen
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
40
Paper Quality improvement of summative radiological skills assessment with digital multi-slice images in medical education.
Cécille Ravesloot • Marieke van der Schaaf • Olle Ten Cate • Jan van Schaik • Anouk van der Gijp • Koen Vincken
Background:Current radiology practice is based on the digital interpretation of volumetric multi-slice (MS)
images (like CT and MR images), however assessment of radiological skills in undergraduate me-dical education is still based on two-dimensional (2D) images. Consequently, the assessment lacks authenticity, which negatively impacts validity. We hypothesized that the quality of assessing ra-diological skills can be increased using MS images.
Methods:In 2010 322 medical students, trained with MS images took a digital radiology test (Version A
and B), containing eleven 2D and ten MS image items (CT-scans). Participants filled out a ques-tionnaire concerning authenticity of the assessment as an indication for validity. Mean scores and reliabilities (Cronbach’s alpha) of the 2D and MS image subtests were compared.
Results:Cronbach’s alphas (corrected with Spearman-Brown formule) on 2D image items (k = 40) were
.52 (Version A), and .68 (Version B) and of MS image items (k = 40) were .79 (Version A), and .78 (Version B). This difference in reliability estimate between MS and 2D image items was significant for Version A.
Scores on MS image items (M 8.0 SD = 1.5; M 7.6 SD 1.5) were significantly lower than scores on 2D image items (M 9.7 SD = 1.1; M 9.0 SD 1.3) (t (df138; 142) = 1.7; 1.5, p < .001; r = .24; .18). Assessment based on multi-slice images was considered more authentic (t (271) = -3.81, p < .001).
Conclusion:According to the participants, assessment with MS images increases authenticity, which can
contribute to validity. Reliabilities were higher for items with MS images compared to items with 2D images. This might indicate that MS image items are more difficult than 2D image items, and therefore more discriminating. Further research with larger tests and studies on participants’ cog-nitive processes involved are necessary.
Keynote Main conference room - Olympia Bourse BallroomUp or Out and the Joy of Lifelong Learning.
Menucha Birenbaum
The presentation offers a conceptualization of assessment culture (AC) in school that stems from complexity thinking. It highlights two complex systems that are nested in the AC system, na-mely classroom learning and teacher professional learning, and points to fractal relations among the three systems and to the recursive interactions whereby they influence each other in a co-evolutionary way. It then depicts the nature of AC and the mindset it reflects, as contrasted to that of a testing culture. Consequences with regard to school locus of control, regulation, and adapta-
14h-15h30: Main conference room - Olympia Bourse Ballroom
11h-12h30: Breakout room 1 Yen
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
41
tion to external demands (accountability) are addressed, and recommendations for fostering and researching an AC are offered.
SymposiumReconceptualising Feedback in Higher Education.
This symposium uses the lens of assessment feedback to examine how to better support stu-dents’ learning. Feedback is an essential part of learning but is often regarded as an end product rather than part of an on-going cycle of learning. Its generation is also largely assumed to be the responsibility of tutors rather than the learner or their peers. By taking a different perspective on feedback we challenge these assumptions, beliefs and resulting practices. Drawing on Feedback: an Agenda for Change, drawn up by a group of international experts (researchers and practitio-ners) at an ASKe event in 2009 the symposium questions the current notion of feedback as pro-duct. It proposes the reconceptualisation and repositioning of feedback within a more coherent paradigm of student learning and assessment which recognises the processual, relational and dia-logic dimensions of effective feedback (e.g. Nicol, 2009) as well as its role in developing students’ evaluative abilities (Boud, 2006).
Largely using theoretical arguments the papers address the need for change and ways in which it can be realised. The symposium will be framed by a paper setting out the ideas and rationale of the Agenda and its calls for change. This is complemented by a paper that explores the nature of feedback through its multiple definitions and the role of self assessment in relation to satisfying the meanings of those definitions in practical terms. The third paper looks at feedback and the learning community, examining the perspectives of staff and students and goes on to develop a process model of self assessment.
In particular these papers illustrate the close relationship between feedback and learning and make arguments to reposition feedback centrally in the process of learning rather than an isolated and final product.
Symposium Chair: Margaret Price Symposium Discussant: David Carless
Margaret Price • Karen Handley • Berry O’Donovan • Chris Rust Jill MillarEffective feedback is a crucial element of assessment and has great potential to enhance lear-
ning. Yet many current practices across Higher Education do not fulfill this potential. Shortcomings in feedback are evident in student dissatisfaction with existing approaches. This paper addresses the growing impetus for change and proposes that to be effective change must be fundamental, involving a reconceptualisation of feedback.
Responding to this impetus a group of international researchers and experts in the field (known as the Osney Grange group, OGG) came together to critically examine and discuss the theory and practice of feedback. They concluded that current practices in Higher Education are often not fit for purpose being largely founded on myths, misconceptions and mistaken assumptions that undermine student learning. For example, one persistent myth is that tutors are the only ones
16h-17h30: Main conference room - Olympia Bourse Ballroom
Paper 1.- Assessment Feedback: Agenda for Change.
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
42
who can give feedback and this largely precludes the use of self and peer assessment that would enable student to develop evaluation skills, a key graduate attribute. Furthermore, current prac-tice is over-reliant on written, unidirectional, tutor-delivered feedback which discourages discus-sion about the meaning and application of feedback. There is a need for stakeholders to re-think the role and purpose of feedback and consequently the OGG developed Feedback: an Agenda for Change (available at http://www.brookes.ac.uk/aske/OsneyGrangeGroup).
The Agenda seeks to promote a re-conceptualisation of feedback from product to process and, in doing so, recognises the importance of its relational and dialogic nature. Feedback needs to have a long-term relational dimension and be supported by dialogue with tutors and peers hel-ping students to better interpret feedback, which is often ambiguous. The Agenda identifies the importance of students ability to use varied sources of feedback and evaluate their own work and that of others. This paper presents, explains and justifies the clauses of Agenda for Change.
Maddalena TarasDifferent educational contexts have resulted in diverse assessment developments in both
theory and practice. Higher education is self-contained and accredits, assesses and awards its own qualifications giving it overall control of formative and summative assessments. The compulsory sector on the other hand, has mainly externally accredited, assessed and awarded qualification with some internally assessed and moderated course-work and therefore sees most of its summa-tive assessments for accredited work out of its immediate control.
Therefore, over time, different communities of practice have developed strategies to support and understand their own contexts and these are manifest in how discourses, definitions and processes are understood and presented.
A superficial commonality of terminology is misleading and potentially dangerous because it is easy to assume that we are speaking the same language when in reality we are referring to di-ffering conceptualisations and epistemologies. This paper examines three aspects of assessment across educational sectors: firstly, definitions of feedback, secondly the relationship between fee-dback and formative assessment and thirdly, how feedback is construed in the five self-assessment models identified by Taras (2010). Feedback is an aspect of assessment which reflects clearly the link between learning, teaching and assessment. How it is construed highlights the roles of the protagonists and how they interact and relate in order to support institutional learning.
The internationalisation of communications through publications, conferences and personal contacts has dissolved barriers across educational sectors requiring an understanding of discour-ses and practices. Transfer of ideas and practices are inevitable and therefore a clear understanding of how theories and practices are context specific is necessary in order to avoid misunderstanding and misuses of work across sectors. By understanding the aspects which are context-specific it is possible to transfer innovative thinking and practices, adapt them to new contexts and maximise efficiency of transfer (Taras 2008).
Paper 2.- Feedback on Feedback: uncrossing wires across sectors.
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
43
Stephen Merry • Paul Orsmond • Karen Handley
Paper Academic assessment centers and student competencies: Reliability, construct, consequential and predictive validity.
Rutger KappeThe paper reports on the use of assessment centers (AC) in higher education. Students (N = 137)
in the human resource management program participated in two AC’s; once at the end of their first year and then again at the end of their third year of study. They were assessed on nine compe-tencies. In addition overall GPA as well as five specific GPA measures; lectures, skills training, pro-jects, internships, thesis were available. Years later, after graduating from college, students were contacted and asked to provide self-ratings of career success.
Results show that, a) competence were measured with high levels of inter-rater reliability, b) the nine competencies could be collapsed into three competency dimensions and an overall factor,
The purpose of this presentation is to explore, within the context of social learning, the conse-quences of feedback practices in Higher Education and to suggest ways feedback can be imple-mented in terms of enhancing student learning outside the formal curriculum.
It has been shown that students develop communities of practice that function in accordan-ce within Wenger’s conceptual framework and particular patterns of participation are displayed. Consequently, an awareness of community practice would seem to be a necessary condition for understanding the student learning that results from tutor feedback. Tutor feedback needs to be reconceptualised, being seen not as communication between an individual tutor and an in-dividual student, but as communication between an academic community and a student com-munity. Translation of language occurs with student communities, perhaps, reinterpreting the intended messages of tutors. Students also engage in peer-assessment through their ongoing practice with others who participate differently in the community, and distributed cognition and self-assessment practices emerge from the peer feedback generated in this way. Each community member negotiates meaning in their individual ways.
Understanding student community practice is also important because it may provide opportu-nities for tutors to enrich the student learning that naturally occurs outside the overt curriculum. If feedback is seen in its broadest sense as encompassing peer and tutor discussions, focussed dialogic teaching practices provide opportunities for situated learning within the curriculum such that common understandings are generated which serve to minimise differences between aca-demic and student communities. Research informed teaching, exemplars and the GOALS process may be suitable vehicles such classroom dialogue. Tutors are also, thorough suitably designed feedback practice, able to enrich and develop students’ self-assessment qualities thus enhancing their learning abilities.
Paper 3.- Students’ social learning and its implications for feedback.
16h-17h30: Breakout room 1 Euro
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
44
c) students received higher AC ratings in their third year, d) the correlation between GPA and AC ratings was significantly higher for third, compared to first, year students, e) AC ratings showed the highest correlation with ratings for internships,
and f ) students perceived the AC experience as being helpful to their learning. Concerning the predictive validity of the academic AC the findings show that competency ratings were the best predictor of early career success, even after holding GPA constant, however the reverse was not true. Whereas AC’s measure broader competencies that are learned throughout the curriculum, traditional exams measure short term, task or course related performance. Because AC’s can be cost effective, educational institutions should consider using existing exams and AC’s alongside one another other.
PaperSchool quality evaluation: students’ points of view.
Valentina Grion • Séverine ColinetAssessing school quality at a national level is a current policy fashion (Broadfoot, 2009) in most
European countries and, at the same time, is a subject widely discussed by researchers. Today, many assume a critical position toward the evaluation systems linked to the Measurement Pa-radigm (Broadfoot, 2009) and highlight its limits and pernicious effects (The Cambridge Primary Review, 2009; Stobart, 2008). They claim to employ strategies to remedy deficiencies of the exces-sively pervading use of performance testing. This might be interpreted as a need to use a greater diversity of tools and multiple sources of data, in order to obtain an effective system of evaluation in schools.
Some authors affirm the importance of measuring client satisfaction and perceptions (Elliot&Sammons, 2000); Walford (2002) asserts that there are two types of client whose voices have
not been listened to sufficiently, the first includes young people who experience school on a day-to-day basis.
In this framework, we are conducting an empirical work to verify the possibility of establishing a set of evaluative indicators, based on students’ representations of “a good school”.
In the first phase, a purposive sample of about 150 students was chosen from a French andan Italian middle school. During normal class time, participants answered a written open ques-
tion which asked them to define “a good school”. Using a bottom-up approach, we conducted a content
analysis of the texts, in order to identify emergent categories of content and to generate meanings from the students’ answers.
The results should highlight the similarities and specificities of each country in order to findcommon indicators for evaluating the quality of schools in a transnational perspective.Although our analyses are at a very early stage, we think there are some interesting aspects to
be discussed.
16h-17h30: Breakout room 1 Euro
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
45
Paper Do different assessment routes (linear vs. modular) prepare students, in the same way, for further study?
Carmen Vidal RodeiroThe modularisation of GCSEs (qualifications taken by students aged 14-16 in England) has cau-
sed considerable controversy since its introduction. There are a number of different perspecti-ves relating to the modular vs. linear assessment debate. Firstly, there are those who believe that modular assessment could lead to lack of coherence and fragmentation of learning as students have little time for reflection, skill development and knowledge consolidation. Secondly, modular assessment might not provide opportunities for deep learning and might, instead, encourage a climate of cramming. Thirdly, the increased assessment load might encourage students to spend more time revising for the modular exams, rather than simply benefiting from the learning expe-rience. Fourthly, there is the view that re-sitting modules may be lowering examination standards, including ‘teaching to the test’ time heightened at the expense of deeper learning.
Based on the above issues, teachers expressed concerns about modular students being less well equipped for the transition from GCSE to further study (e.g. A-levels, qualifications taken by 18 year-olds) than their linear counterparts. This study set out to investigate whether modular cour-ses are good preparation for further study. The focus was on the impact of the GCSE assessment route on the uptake and performance in three A-levels: English, mathematics and ICT.
The research showed that students following a linear assessment route at GCSE were more likely to continue to study mathematics at A-level than those who followed a modular route. Con-versely, linear students were less likely to progress to A-level in ICT. There were no differences between linear and modular students in the uptake of English at A-level.
Once attainment at GCSE was taken into account, the assessment route only affected perfor-mance in mathematics, with students following a linear route at GCSE performing better at A-level than those following a modular one.
PaperEarly literacy assessment and Intervention.
Julie Brosnan • Elisabeth Scherz • Jesse Butler • Olive Healy • Kendra Brooks-Rickards
A Response to Intervention Framework is an all-inclusive early detection and prevention stra-tegy designed to identify and support struggling readers at the first sign of difficulty (Gersten et al. 2009). RTI frameworks operate through a multi-tiered approach; this study investigates the effectiveness of a Tier 2 intervention designed to support the development of Senior Infant (kin-dergarten) student’s early reading skills. The Tier 2 Intervention combines Children’s Progress Aca-demic Assessment (CPAA) as the universal screener to identify the pool of Tier 2 students, with an intervention defined as precision teaching targeting the component skills related to reading (e.g. letter sounds, letter identification, initial sounds, final sounds etc.). Thus the study focuses on an
16h-17h30: Breakout room 1 Euro
16h-17h30: Breakout room 1 Sterling
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
46
Paper Effects of Self-Paced Learning on Achievement in Sentence Level Work in Year 5.
Mary Sheard • Bette ChambersA development in the use of learner response devices (LRDs)allows pupils to answer questions
at their own pace, providing instantaneous feedback to the pupils and their teachers about the pupils’ understanding of the concepts just taught. Teachers should then be able to provide diffe-rentiated support to pupils, improving their learning. A small-scale randomised evaluation of the self-paced learning strategy (SPL) was conducted to determine if this form of technology–sup-ported formative feedback increases pupil learning. Forty-two Year 5 classes in primary schools in the North of England were randomly assigned to experimental (21 classes) or control (21 classes) conditions for a 12-week trial period during the Spring term, 2012. An additional 16 Year 5 classes who were already experienced in using LRDs participated as a matched comparison user group, so providing a second experimental group. Learning Clip, an interactive resource developer, pro-vided 40 question sets with a grammar focus for the experimental groups to implement in their literacy lessons during the research period.
Equivalent pre- and post-tests reflecting the Year 5 sentence-level content were the measures used. Training and support in SPL using ActiveExpression learner response devices was delivered by Learning Clip personnel to teachers in the experimental groups in their schools and reinforced by ongoing support.
Results from a similar previous study in mathematics showed no statistically significant diffe-rence between pupils’ scores at pre-test and a statistically significant difference in their post-test scores. In reporting findings from this more recent study, the present paper draws parallels with the earler findings and makes recommendations for maximising the potential for technology-supported self-paced learning in increasing primary children’s grammatical achievement. In addi-tion, the paper suggests how the empirical analyses relates to an extended theoretical model of formative asassessment in the context of educational effectiveness.
assessment-intervention-evaluation cycle within the applied setting.The study will run for 8 weeks, with approximately 15 ‘senior infant’ (kindergarten) students
ranging in age 5-6.10 years serving as participants. The setting is a primary school located in a rural area in the west of Ireland. All 15 senior infant students will be administered 4 different types of assessments related to early literacy skills (Children’s Progress Academic Assessment; Dynamic Indicators of Basic Early Literacy Skills; Curriculum Based Measures; The York Assessment for Rea-ding Comprehension). Those Senior Infant students who score ≤2 on Children’s Progress Acade-mic Assessment subtests of Phonics and Writing, and Phonemic Awareness will be selected for Tier 2 intervention for the duration of the study. Each student will receive 15-20 minutes daily of targeted precision teaching procedures during the school week. Tier 2 intervention effectiveness will be investigated within a multiple baseline experimental design. Data will be analysed through visual inspection and analysis of the data; estimation of effect sizes; and calculation of ratio gains for outcome measures. Results will be available in March 2012.
16h-17h30: Breakout room 1 Sterling
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
47
PaperPractical problems in the assessment of text comprehension.
David Hidalgo • Jesús Alonso-TapiaAssessment influences learning activities. So, it is necessary to investigate whether teachers as-
sess knowledge and competence adequately. In this study the objective was to identify strengths and shortcomings in teachers’ ways of assessing text comprehension. With this purpose, after es-tablishing the psychological models from which ideas on text comprehension assessment should be evaluated, a sample of 111 Spanish Primary School teachers were asked to design assessment tasks in relation to two texts of different structure. The tasks should be adequate for deciding the kind of help to give the students to enhance their text comprehension. The lack of regular as-sessment patterns emerging from results showed that teachers’ practice does not correspond to any shared model. Moreover, results also showed specific strengths and shortcomings of teachers’ ideas on text comprehension assessment, as well as the degree in which the kind of text influen-ces assessment practices. Implications for designing teachers’ training on students’ assessment are discussed.
16h-17h30: Breakout room 1 Sterling
Paper The impact of a dedicated assessment course on developing teacher education students’ assessment capability.
Mary Hill • Alison Gilmore • Lisa Smith • Bronwen CowieThe primary aim of the research was to explore how primary teacher education students in three-
year programmes across four universities in New Zealand develop their assessment knowledge, beliefs and practices (assessment capability) during their teacher education programmes. In all four universities the assessment curriculum is embedded within curriculum and professional practice courses as well as practicum experiences in schools. However, in the second year of stu-dy, two universities provide a dedicated assessment course, as well. Participants were the 2010 entering cohorts (n = 711) at the four universities. The second-year students in two of the univer-sities with the assessment course (n = 289) were compared with those in the universities that did not have a dedicated assessment course (n=282). Data were collected through a Beliefs About Assessment questionnaire administered at the beginning of year one and at the end of year two. The questionnaire contained six open-ended questions and 46 Likert-type questions based on a conceptual framework of assessment capability developed from a position as naïve student to emerging professional.
Analyses of the first year students’ assessment knowledge, beliefs, and practices established that their views about assessment were ‘naïve and largely represented their own experiences of being assessed as students themselves. There were no differences between universities in ente-ring students’ views. However, analyses of students’ views after their second year of study revea-led emerging understandings of assessment consistent with being a teacher. In responses to the Likert items, there appeared to be little impact of students taking a dedicated assessment course. However, qualitative analyses of the open-ended questions revealed differences in the views held by the two groups of students.
16h-17h30: Breakout room 1 Dollar
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
48
Paper Student and Teacher Educator Perceptions of Becoming Assessment-Capable Teachers.
Lisa Smith • Bronwen Cowie • Alison Gilmore • Mary HillThe primary aim of the research was to explore what primary teacher education students in a
three-year programme across four universities in New Zealand believe and understand about as-sessment at entry, during, and at exit from their programmes. Participants were the 2010 entering cohorts (n = 711) at four universities. Third-year exiting students (n = 289) at the four universities provided a comparison group. In addition, at each university, n = 6-10 entering student partici-pants took part in focus groups and n = 8-12 teacher educators took part in professional conver-sations. Data were collected through a “Beliefs About Assessment” questionnaire, semi-structured protocols for the student focus groups, and semi-structured protocols for the teacher educator conversations. Analyses of the questionnaire items comparing the entering cohort to the exiting cohort indicated that significant changes to beliefs about assessment occurred over the course of 3 years. There was progress from a naïve view that assessment is mainly formal and summative in nature, toward an emerging professional view that highlighted the need to understand the role of children in assessment for learning. In the teacher educator conversations, five themes emerged: an emphasis on teaching about formative assessment, noting the need to shift students’ thinking from summative to “next steps,” the need for students to understand using multiple sources of evidence, learning how to make judgments, and having strong curriculum knowledge. The stu-dent focus groups showed beginning understandings of formative assessment and brought out
Paper The electronic portfolio as an instrument for the development of teacher competences in pre-service teaching practice.
Katrien Struyven • Yves Blieck • Veronique De RoeckDuring recent years teaching practice has gained importance as a significant period of learning
for the development of teacher competences. These teacher competences are regarded as the possession and development of sufficient skills, knowledge, appropriate attitudes and experience for the successful performance of one’s professional roles as a teacher (McNamara, 1992). Hen-ce, many teacher education institutions are aiming at the development of teacher competences (also) during teaching practice. A popular instrument to achieve this aim and to assess the develo-pment of teacher competences is the electronic portfolio. This research study investigates the use and effects of a competence-based electronic portfolio for the assessment of teacher competen-ces during teaching practice in a Flemish academic teacher education program. Therefore, ques-tionnaires and semi-structured interviews were administered to three parties, student teachers (N=41), teacher trainers (N=25) and mentors (N=46). Results show that the portfolio proofs to be a useful instrument for the assessment of teacher competences according to teacher trainers and mentors. Student teachers, in contrast, do not belief that the portfolio achieves this aim. In their views, the portfolio is unable to reflect and demonstrate their teacher competences sufficiently. Why that is the case, is the central focus of this paper.
16h-17h30: Breakout room 1 Dollar
16h-17h30: Breakout room 1 Dollar
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
49
the impact of practicum experiences on student thinking and practice. The discussion focuses on what teacher educators need to be aware of in providing for students in the early stages of their education.
Paper A Validity Framework for Assessments Designed to SupportInstruction.
James Pellegrino • Louis DiBello • Susan GoldmanThe overarching goal of the work we propose to report involves implementation of a compre-
hensive, multidisciplinary approach to the design and validation of assessments that are intended to function close to instruction, i.e., assessments designed for both summative and formative pur-poses. The goal of the work is to transform how such assessments can be used in STEM classrooms and enhance their instructional effectiveness and impact on student learning. Our multidiscipli-nary approach integrates disciplinary knowledge of STEM content, experience in STEM area tea-ching and learning, cognitive science, educational measurement, and psychometric modeling. The cases we are working on include assessments embedded in K-8 mathematics curricula, tech-nology-based diagnostic assessments for 6-12 physical science, and concept inventories used in undergraduate engineering courses.
The project’s research questions and associated data collection flow from a framework we have developed for studying the validity of assessments that are intended for use in instructional con-texts. The validity framework describes three major components labeled as: Cognitive, Instructio-nal, and Inferential aspects of validity. We will provide the rationale for and descriptions of each aspect of validity and also describe the forms of evidence that one needs to collect to support a comprehensive validity argument regarding the quality and utility of any given assessment desig-ned to function close to instructional settings.
We will show how existing assessments designed to function close to instruction vary in the extent to which evidence is available for the different aspects of validity and how the nature of the evidence is a function of design features of the specific assessments. The project’s long-range goal is to implement a research- and theory-based framework for the development and use of assessment resources in educational contexts such that well-designed assessments can serve as integral components of the instructional materials and the overall learning environment.
Paper Deducing ‘learning outcomes in action’ from lecturers’ assessment talk.
Susan Orr • Susan BloxhamLearning outcomes are a taken for granted part of the higher education assessment landscape.
In the UK, the national organisation that assures standards and quality in higher education (the QAA) states that institutions need to provide for the ‘effective and appropriate measurement of students’ achievement of intended learning outcomes’ (QAA 2006:31). Given the centrality of lear-ning outcomes it is noteworthy that several studies report that, in spite of QAA guidance, lecturers
16h-17h30: Breakout room 1 Yen
16h-17h30: Breakout room 1 Yen
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
50
Paper Developing ‘assessment capable’ beginning secondary teachers in New Zealand.
Frances Edwards • Beverley CooperPre-service teacher educators work with student teachers to help them develop their own
knowledge, beliefs and skills about assessment throughout their teacher education programmes. During their study and practicum experiences student teachers develop a level of assessment capability. This capability is particularly important in the New Zealand context, where secondary teachers are responsible for the design, choice and marking of “internal” assessment tasks used to assess senior students, the results of which contribute directly to the students’ school leaving cre-dential - a standards-based qualification, National Certificate of Educational Achievement (NCEA).
This paper reports on part of a 4-year study which aims to address gaps in our knowledge about secondary conjoint student teachers’ knowledge and beliefs about assessment, and how these change and develop over time. The study involves a systematic investigation of how their ideas and practices develop over the course of a four-year secondary conjoint teacher education pro-gramme.
This was a mixed methods study where students were surveyed yearly, using a questionnaire. Eight students participated in focus group discussions each year, and kept journals for the dura-tion of their teacher education programme.
The presentation will describe findings on the range of views about assessment held by se-condary conjoint pre-service teachers at entry into their teacher education programme and as they move through the programme. Particular factors that contribute to their understandings and practices will be discussed.
Conclusions will be drawn about the implications of the findings and how learning about as-
do not always anchor their assessment decisions exclusively on the basis of assessing student work against the learning outcomes (see for example, Hand and Clewes 2000, Hawe 2003 and Orr 2007). These studies appear to be premised on the expectation that student work would be asses-sed against the documented learning outcomes and they explore the extent to which lecturers’ assessment practices depart from this ideal.
In this study we explore the learning outcomes associated with art and design assessment. However, crucially, we do not anchor this study in the documented learning outcomes found in the course literature. The research reported on here has a different starting point. Art and design lecturers’ assessment talk becomes a key site for carrying out a grounded, bottom up analysis of emergent ‘learning outcomes in action’ used in live assessment contexts. Thus the question is; can we deduce the learning outcomes from the assessment talk of a small group of art and design lecturers?
The authors audio-recorded lecturers while they marked artwork alone (using thinking aloud protocols) and audio-recorded meetings in which lecturers agreed marks in two English universi-ties. The analysis of the transcripts led to the identification of three interrelated learning outcomes in action. These ‘learning outcomes in action’ are used as a lens to explore and challenge the way we understand documented learning outcomes in HE.
16h-17h30: Breakout room 1 Yen
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
51
sessment could be further enhanced over the course of a teacher education programme that involves the simultaneous development of pedagogical and curriculum subject (discipline) knowledge.
Friday 31 augustInvited
Symposium Assessment for learning: varying perspectives with a focus onlearners.
Assessment for Learning (AfL) is a broad concept. In fact, it is often argued that it is ill-defined. Nevertheless it is one of the most active and widespread areas of practice development in as-sessment. The papers in the symposium report on studies using different theoretical perspectives and empirical approaches to examining AfL, bringing clearly-defined frameworks and definitions to bear. Most research on AfL addresses: principles and design; assessment interventions in the curriculum; and impacts of AfL interventions on learning and achievement. A distinctive aspect of the research reported here is the focus on students Although an active learner is central to AfL, the learner perspective has not been well explored. One of the emergent perspectives is that of the student, within a social and institutional context explored using cultural historical activity theory. The concept of assessment literacy is developed in one of the studies, proposing ways in which students need to be prepared if they are to be successful in AfL contexts. As a result, this leads to an assessment curriculum which is centred on long term goals(Boud & Falchikov, 2007). Another paper reports on two practitioner research studies. One identifies the scope and significance of students’ communities of practice beyond the curriculum. A second focuses on students’ construc-tions of AfL noting substantial difference in emphasis when compared to teacher constructions. Students foregrounded dialogue, social interactions and informal feedback rather than specific interventions such training for and participation in self and peer assessment. The implications of the research for practice, particularly in higher and professional education, will be discussed.
9h-10h30: Main conference room - Olympia Bourse Ballroom
Anton HavnesThis paper takes as the starting point that one of the challenges that need to be addressed in
the development of assessment for learning is the theoretical grounding of formative assessment. Research on formative assessment is often associated with constructivism and the notions of in-dependent learning and the self-regulated learner (Sadler 1987, 2010; Nicol, 2009; Wiliam, 2011). However, this emphasis on self-regulation, based on internal control of cognitive processes, is countered by an emphasis on student-teacher dialogue, peer assessment and social interaction
Symposium Chair: Dominique Sluijsmans Symposium Discussant: Liz McDowell
Paper 1.- The role of assessment in learning - A Cultural Historical Activity Theoryperspective.
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
52
Margaret Price • Berry O’Donovan • Chris Rust • Karen Handley • Birgit den OuterEffective assessment for learning means that students need a more sophisticated understan-
ding of how assessment works and how it relates to their learning. This paper explores the emer-ging concept of assessment literacy to negotiate the changing complexities of assessment in con-temporary higher education. The term ‘literacy’ has traditionally been used in the context of skills and associated with attributes such as fluency, competence, or expertise. However, it also has more subtle connotations relating to ‘gateway’ or ‘threshold’ skills which, once mastered, allow access, not only to further learning and knowledge, but also to academic communities as the custodians of academic standards, practices, customs and culture. On the basis of a programme of empirical research and practice development in assessment for learning, it is proposed that to be fully literate, students need to recognise and understand the many aspects of assessment as well as the relationship between them. It is argued that there is a reiterative relationship between assessment for learning and student assessment literacy, and for current assessment for learning initiatives to take full effect, the development of assessment literacy is needed. It is proposed that Higher Education practice steers away from an ad-hoc, and therefore largely, ineffectual approach to the development of assessment literacy and instead, that students are engaged in a program-me of purposive development activities which have been shown to be effective.
(e.g. Nicol, 2010). There is a mix of a self-regulated learning approach and a more socio-cultural approach building on Vygotsky’s theorising of the role of social interaction in learning and more recent developments in Cultural Historical Activity Theory (CHAT) emphasising the embedded-ness of assessment and learning in social, historical and institutional context. Another challenge is related to the context in which assessment for learning and feedback is framed. In this paper formative assessment will be discussed in the context of some particular challenges in professio-nal education.
I will discuss formative assessment in a CHAT perspective by drawing attention to three levels of analysis that need to be included in cultural historical analysis:
•assessment as component in the teaching-learning relationship, emphasising the role of as-sessment in the zone of proximal development (zpd)•the embeddedness of learning and assessment in an activity system which provides affor-
dances and constraints on educational practices•the interrelatedness between education and the ambient world, in this case professional edu-
cation and professional practice The paper contributes to the discourse of the concept of formative assessment and the relation-
ship between assessment, learning and teaching. It also raised a debate on the particular challen-ges associated with developing formative assessment in professional education.
Kay Sambell • Paul OrsmondIn Assessment for Learning (AfL) the student is regarded as active in the construction of learning
Paper 2.- Assessment literacy - a perspective on the student role in assessment for learning.
Paper 3.- Students’ constructions of assessment for learning: within and beyond the cur-riculum.
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
53
and assessment, so it is important that the student perspective and experience is researched. Two close-up investigations of university students’ constructions and practices in relation to AfL are reported. Both studies reveal aspects which are often neglected when AfL is viewed solely as a teacher-led, curriculum-led practice. It is identified that students create their own ‘hidden curricu-lum’ in response to assessment for learning. This is not a negative phenomenon but builds on the roles that we expect learners to play. One of the empirical studies used the conceptual framework of communities of practice. It revealed the extent and quality of student communities of practi-ce which are largely overlooked in favour of promoting participation in academic communities. Student communities of varying kinds, established quite independently of teachers, were shown to fulfil functions including: sharing exemplars of academic work, testing knowledge, motivation and developing identities. In interaction with students, tutors should be aware that they are not just speaking to individuals but to another community that has its own rules and practices. In another empirical study, students produced guides to assessment for learning for a student au-dience. Content analysis of students’ guides showed that they tended to interpret and reframe the assessment for learning agenda in different ways and with noticeably different emphases than the ones which are prominent in the texts written for and by academics. They placed markedly heavier emphasis on student participation in teaching, learning and feedback practices in general and particularly foregrounded dialogue, social interactions and informal feedback. They remained relatively silent about interventions designed to build students’ appraisal experiences. Implica-tions for assessment practice are discussed.
Symposium Teacher and student roles during feedback in interaction.
In education it is common practice that students have feedback dialogues with their supervi-sors or teachers. A feedback dialogue is a conversation in which students receive verbal feedback from their teacher and exchange information about their performances and learning process and performances with the teacher in order to reduce the gap between their performances and the intended learning goals. Being able to reduce ‘the gap’ requires that students: a) have a concept of the standard of performance, comparable to that of the teacher; (b) continuously compare the actual level of performance with the standard, which implies reflective thinking activities like analyzing and evaluating;(c) engage in appropriate actions to close the gap, which means that students have a repertoire of suitable strategies to draw from (Sadler, 2010). In such a conversation feedback can be seen from an interpersonal (social), cognitive and affective perspective. In this symposium these perspectives are key to research oral feedback situations between teachers and students in secondary education, higher education and vocational education. The symposium encloses four papers: 1) A position paper that discusses the value of the feedback concept from a positive psychological, interpersonal and affective perspective (Voerman et al.); 2) A research paper regarding interaction in feedback dialogues to stimulate students’ reflective thinking (Van der Schaaf); 3) A research paper about the relation between teacher-student interpersonal beha-vior and characteristics of feedback (De Kleijn et al.); 4) A research paper in which an instrument is
Symposium Chair: Marieke van der Schaaf Symposium Discussant: D. Carless9h-10h30: Breakout room 1 Euro
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
54
Lia Voerman • Paulien C. Meijer • Fred Korthagen • Robert Jan SimonsIn this paper we first present a brief summary of the theory on teacher feedback. Than we intro-
duce some issues that until now seem to have been overlooked or undervalued in most literature on feedback. We add the concept of progress feedback to the conception of feedback as feedback on the discrepancy of an initial level of performance and a goal. Progress feedback is described as feedback comparing an initial level of performance with a current level of performance. We make a distinction between praise and feedback on the self and describe both concepts separately. Praise is described as non specific positive feedback. Building on research result from positive psychologists we propose to provide feedback on character strengths as a way to provide fee-dback on the self. Research form positive psychologists showed the impact of emotion on lear-ning. Elaborating on these research findings, we discuss the impact that feedback has on emotion and thus on learning. Hence we stress the importance of taking into account the emotion that is evoked by feedback, while providing feedback. Implications for teaching practice in schools entail recommendations for teachers how to provide effective feedback.
Marieke van der Schaaf How can feedback dialogues stimulate students’ reflective thinking? This study aims to inves-
tigate: (1) the effects of feedback dialogues between teachers and students on students’ percep-tions of teacher feedback and (2) the relation between features of feedback dialogues and stu-dents’ thinking activities as part of reflective thinking. A quasi-experiment was carried out in which seventy-two secondary education students received written teacher feedback. Half of them had an additional feedback dialogue with their teacher. The latter group perceived teachers’ feedback as more useful than the group without a dialogue. The feedback dialogues could be clustered in two groups that varied in the number of segments containing teacher and student interaction. The number of segments containing interaction was positively correlated with students’ use of thinking activities.
constructed to measure students’ interpersonal behavior within teacher-student feedback situa-tions (Prins & Mainhard).
Renske de Kleijn • Paulien Meijer • Albert Pilot • Mieke BrekelmansIn master’s thesis supervision one of the most important tasks of supervisors is to provide cons-
tructive feedback, often in feedback dialogues. As previous research has indicated the importance of a feedback perceptions and an interpersonal perspective on these feedback dialogues, both perspectives are investigated in the present study: how are student interpersonal and feedback perceptions related and to what extent can a combination predict student satisfaction (SS) and perceived supervisor contribution to learning (PSCL)?
Paper 1.- Some issues on teacher feedback.
Paper 2.- Interaction in Feedback Dialogues to Stimulate Students’ Reflective Thinking.
Paper 3.- Interpersonal and feedback perceptions in master’s thesis supervision.
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
55
The results of this study will provide insight in the relation between two perspectives on fee-dback dialogues (feedback and interpersonal perceptions) and will yield insight concerning to what extent these perspectives overlap and/or explain distinct variance in SS and PSCL. Feedback perspective
Based on the findings of two review studies concerning feedback (Hattie & Timperley, 2007; Shute, 2007) feedback perceptions are conceptualized in terms of perceptions of focus, elabora-tion, and goal-relatedness.Interpersonal perspective
Perceptions of the supervisor interpersonal behavior were conceptualized based on the model for interpersonal teacher behavior in which the two interpersonal dimensions control and affilia-tion are discerned.
In total 1016 master students completed the questionnaires.Instruments:
•Feedback perceptions: QFP (de Kleijn et al., submitted)•Interpersonal perceptions: QSDI (Mainhard et al., 2009)•SS: two items (r=0.81; de Kleijn et al., in press)•PSCL: five items (α=0.89; de Kleijn et al., in press)
Correlation analyses, cluster analysis and ANOVAs and regression analyses were applied.The results of this study suggest that both a feedback perceptions perspective and an interper-
sonal perspective on feedback dialogues in master’s thesis supervision provide partly overlapping but also distinct information as it was found that all subscales are related, but that also all scales except for focus were significant predictors for SS and PSCL. Affiliation was found to be the most strong predictor.
Frans Prins • Tim MainhardNicol (2010) and Carless et al. (2011) argue that feedback can be more effective when it is pro-
vided during a dialogue between teacher and student instead of by monologues from teacher to student. During a feedback dialogue, teacher and student may adopt different roles (e.g., Hadwin et al., 2005) and they may adapt their behavior to the behavior of the other person. Such inter-personal dynamics of feedback dialogues have been shown to be related to the effectiveness of feedback. It is therefore important to examine how interpersonal teacher behavior and interper-sonal student behavior during feedback dialogues affect each other. However, there is no instru-ment available to measure supervisors´ interpersonal perceptions of student behavior. This paper describes the construction of a questionnaire that taps supervisors´ interpersonal perceptions of student behavior during feedback dialogues.
Two studies are presented. Study 1 is a Delphi study in which 10 experts rated descriptions of student behavior in feedback dialogues, provided by students and experienced teachers, on sui-tability and interpersonal meaning. The experts also categorized these descriptions on the inter-personal circle. Study 1 resulted in a prototype of a questionnaire on interpersonal perceptions of student behavior in teacher-student feedback dialogues. In study 2, the questionnaire was tested on reliability and validity. Results of study 2 indicated that the questionnaire is a reliable instru-
Paper 4.- Tapping supervisors´ interpersonal perception of student behavior during feedback dialogues: development of a questionnaire.
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
56
ment, but that the reliability of items representing passive student behavior was somewhat lower. Furthermore, interpersonal profiles of, according to supervisors, good, average and ineffective students differed significantly, which indicates the external validity of the questionnaire. Finally, the circumplex structure of the items and scales of the questionnaire indicated internal validity. Thus, the questionnaire in its current form is a reliable and valid instrument that can be used for re-search on the interpersonal aspects of the interplay/dynamics between teacher/supervisors and student during feedback dialogues.
Paper Comparing different item-types in paper-based and computerized environment.
László HülberAltering the measuring valuation processes of technologically based results is not only modi-
fying the medium but the CBA provides such possibilities - compared to the PB testing which can
Paper Innovative Assessment Technologies in Educational Games Designed for Integrating Assessment into Teaching.
Gyongyver Molnar • Brigitta Mikszai-Rethey • Attila Pasztor • Timea MagyarTechnology offers a great deal of new opportunities in educational assessment and vitalizes
testing situation by allowing innovative task presentation, including multimedia, which increa-ses motivation. Instead of providing single indicators, such as a test score, technology-based as-sessment may produce rich, contextualized data sets, which supports the transition from single testing to complex systems of assessments. The purpose of this pilot study is to investigate the opportunities and effectiveness of applying educational games and embedded innovative as-sessment technologies to improve students’ reasoning skills. First and second grade (age 6 to 8) students constituted the experimental group (n=38), whereas the control group consisted of 137 students. The effectiveness of the training was measured with an inductive reasoning test, compri-sing 37 figural, non-verbal items (Cronbach α=.87). Beside test-based data collection, innovative assessment technologies are explored by logging metadata, such as facial expressions and head movement (captured by web camera). There was no significant change in performance between pre- and post-test in the control group (t=1.44, p>.05), while the experimental group managed to achieve significant development in the experimental period (t=-10.65, p<.00), significantly out-performing the control group by more than one standard deviation on the post-test. The effect size of the training program was d=.87 (p<.01). The most frequent facial expressions were those of surprise (31%), happiness (24%) and anger (16%) during the training. Disgust (7%), fear (3%) and sadness (1%) were less frequent. No significant differences were found between genders with regard to the distribution of the different expressed emotions. The distributions of the facial ex-pressions did not show significant relationships with level of reasoning skills. Our findings pro-vided further empirical evidence that innovative assessment technology gives the possibility to integrate assessment into teaching by giving proper assessment tools to monitor both cognitive (e.g. reasoning) and affective (motivation, interest, boredom) processes.
11h-12h30: Main conference room - Olympia Bourse Ballroom
11h-12h30: Main conference room - Olympia Bourse Ballroom
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
57
reform the whole measuring valuation process. For responsible alteration the media effect has to be identified with regards of (1) sample, (2) test characteristics, (3) technical parameters. My ob-jective was to separate those mathematic erudition fielded items which are primarily responsible for the CBA and PP result discrepancy, to collect their mutual characteristics and to type them. I have used the diagnostic measurement data recorded in 2010 in Hungary on CBA (N=943) and PP (N=20400), the testing carried out in the first and sixth grades. I have used those 299 items for item diagnostics, which had already been defined on both the CBA and PP. I started the data analysis with item response theory models, which gave the opportunity to characterize those students who are in different grades but filled out tests based on mutual ability scale, furthermore, to cha-racterize items in different tests on the same scale. I made further analysis with the help of classical test theory methods on the result of the multidimensional analysis items. The results show that the item difficulty correlation in CBA and PP is r= .64, which is significant on the level of p< .01. The item difficulty values are weakly correlating with the item’s character number which are at CBA: r= .32 p< .01, at PP: r= .25 p< .01. There is no significant difference neither at the open ended tests in the PP and CBA results: t=1.18, p= .24; nor at the closed ended tests t=-1.07, p= .28. With applying the helpful or needed graphical elements, there is a significant difference between the CBA and PP results: t=2.22 p< .05.
PaperAssessment Planning for Interactive Case-based Learning Scenario.
Genevieve Gauthier • Jeannine Conway • Rebecca TaylorThe use of cases in teaching offers the potential to meet the challenges of today’s evolving
knowledge, yet the evaluation of case performance poses challenges to current assessment prac-tices. This research explores how expert instructors conceive and plan assessment through the design of interactive computerized cases in pharmacy education. This initial study we explore instructors’ pre-assessment planning for cases they will be using with their students. We describe and discuss reliability issues regarding setting proposed “good” answers for problem solving tasks. Assessment challenges were raised during the development of patient cases for an interactive computerized program called BioWorld. When experts and case creators were asked to perform relatively easy cases using the software, we encountered reliability issues regarding the proposed “good” answers for the cases and the answers the experts actually performed. Experts and case creators could neither reproduce the “proposed” good answers for the cases, nor upon replica-tion of the cases, repeat their first answer to the cases. We suggest that assessment of case-based learning is more complex than simply identifying the “right” answer to a problem and it may be useful to conceptualize the notion of case solving as an ill-defined problem solving performance, as opposed to simply a way to trigger knowledge.
11h-12h30: Main conference room - Olympia Bourse Ballroom
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
58
PaperHow multiple perspectives complicate comparability.
Gill ElliotMultiple perspectives are at the heart of comparability issues and dilemmas. Leaving aside the
issue of comparability over time, if students are permitted to make choices in education, there will be a need for comparability between the different options which are available. In the English edu-cation system there exist high stakes assessments taken by many students at the ages of sixteen and eighteen years. Students have many choices, both about which subjects they take and which particular assessments they choose. The choices they make can impact upon their opportunities for Higher Education entry or employment. An effective means of making comparisons between these different pathways needs to be applied to ensure fairness in a situation where increasing numbers of competing students offer a wide variety of different qualifications in a large number of subjects.
In this paper the author discusses two separate issues relating to multiple perspectives which affect comparability of large scale assessments in England. Firstly, the multiple dimensions that constitute comparability will be explored, and the impact that these dimensions have upon the public understanding of the issue and the effectiveness of solutions which have been attempted will be discussed. Secondly, the diverse perspectives of the different users of the assessments will be investigated, revealing that comparability is not necessarily defined in the same way in all si-tuations.
The way in which comparability is perceived affects both educational policy and practice in England. In drawing conclusions about the effect of multiple perspectives upon the implementa-tion of comparability practice the author will consider some of the evidence from the past as well as the direction of policy in the future.
Paper First results of Hungary´s full cohort longitudinally connectedassessment.
Ildikó BalázsiHungary’s National Assessment of Basic Competencies, an annual full cohort mathematics and
reading literacy assessment, with its latest innovation, the introduction of the measurement ID in 2008, became a longitudinal study. The paper explains the National ABC‘s main features. First, the assessment frameworks and test administration processes are shortly introduced. The scaling and analysis methodology is discussed, the grade- and year-invariant IRT scales and profession levels, weighting and computing algorithms of error terms are described. Then, the paper focuses on the newly developed longitudinal aspects, the analysis of the two year long development of students’ abilities on the measured cognitive domains.
The paper briefly presents the main results of the 2010 and 2011 assessments, focusing on the longitudinal results, the development of students’ abilities during the two years between two assessment points. The relationships of settlement type, school type and socio-economic status
11h-12h30: Breakout room 1 Euro
11h-12h30: Breakout room 1 Euro
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
59
PaperDeveloping a coding scheme for analysing peer feedback messages.
Jan-Willem Strijbos • Bas van Goozen • Frans PrinsIn line with the increased attention for peer assessment research, the peer feedback component
of peer assessment is also more systematically considered. Recent studies that examined the con-tent of peer feedback revealed that the content of peer feedback appears to be a crucial aspect for its effectiveness. However, although several studies analysed the content of peer feedback messages, the coding schemes developed are not systematically tied to a theoretical framework and almost exclusively address cognitive processes. Hence, this study reports the development of a coding scheme for peer feedback messages that moves beyond the cognitive dimension. In addition, the coding scheme heavily draws on the feedback framework developed by Narciss (2008) and distinguishes between ‘feedback function’ and ‘feedback content’. The developed co-ding scheme consisted of five main categories (cognition, meta-cognition, affect, self-efficacy and noncodable) and 24 sub-categories. Cognitive and meta-cognitive processes were further refined with sub-categories for positive and negative verification and five types of elaboration (question, correction, confirmation, suggestion and justification). Since the coding scheme heavily draws on the framework by Narciss (2008) it can be considered theoretically valid. Reliability trails revealed that the coding scheme is sufficiently reliable at the main-category level (cognitive, meta-cogni-tive, affective, self-efficacy and noncodable), as well as the distinction between verification and elaboration. At the sub-category level the reliability is satisfactory, but could be further enhanced in future studies. The coding scheme developed allows for more precise analyses of peer feedback messages and investigating and determining which aspects of a peer feedback message affect the application of peer feedback.
with the results and the development are dealt with. Finally, report types and reported content are discussed: such as online schools’ and maintai-
ners’ reports that provide a multi-aspect review of the results compared to national results and to the results of similar schools in regard the schools type, student intake and initial abilities of students as measured in the two year earlier assessment.
Paper Prompt Feedback by an assessment instrument (Evalcomix) in an Engineering Degree
Daniel Garcia-Almiñana • César Valderrama • Beatriz Amante • María MartínezThe adapting process to the European Higher Education Area (EHEA) gives a different value to
the evaluation process, since it has not only be a certification of that students have learned, but the overall process in which the students can be aware of how and what is learned. This implies that the assessment should be formative as well as summative, for instance, it must be focused to promote or facilitate learning in addition to a normal scoring process. A fast and detailed feedback of learning process is required and allows students to continuous improvement. The teacher fee-
11h-12h30: Breakout room 1 Sterling
11h-12h30: Breakout room 1 Sterling
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
60
dback is not always required; but it is recommended that the students themselves or their part-ners also can offer this feedback. Thus, feedback can be very useful both to motivate and involve students in their own learning process and to accounting for non-lecture time of the students; on the other hand, this may reduce the teaching load especially for large groups. In order to assure that students participation in feedback helps and influences the improving learning process is es-sential to define with clarity the criteria and the objectives of assessment tasks; furthermore they should be used easily and automatically in order to ensure the immediacy of the feedback. One of the main objectives of The GRAPA innovation group at the Universitat Politècnica de Catalunya is provide to the teachers of strategies and assessment tools to facilitate the adaptation to the EHEA. EvalCOMIX is one of these useful tools which have been developed to facilitate feedback in assessment process under compatible structure to the virtual campus (Moodle).
In this work is reported the assessment tools created and used through the EvalCOMIX in the subject of Engineering Projects of Engineering in Industrial Organisation degree (part-time atten-dance course) during the course of 2011-2012.
Kristine Høeg Karlsen • Joakim KarlsenThis paper reports from our work on understanding the complex relationship between types of
teacher feedback and students learning activities. We have applied Karlsen’s (2011; in press) latest work on classification of feedback-types and Krathwohl’s (2002) revisited framework of Bloom’s taxonomy of educational objectives in order to explore this relationship. A range of factors in-fluences what kinds of learning activities students engage in. To single out and investigate the causal relationship between these activities and teacher feedback is challenging. In our research we don’t claim any direct causal relationship between the phenomena, but try to analyse and un-derstand students and teacher’s language and actions together framed by short feedback loops. Findings are based on the analysis of these texts internal references. The teacher’s and the stu-dents in this study frequently refer to specific statements, tasks and topics occurring in the same feedback-loop.
A qualitative approach using ethnographic methods for data collection and analysis was used. Two comparative cases at two faculties in the field of technology at the same university college in Norway were chosen. In both cases students was working on their bachelor thesis group-projects, similar in both content and organisation. The sample consists of 28 students and their six super-visors. The collected data material consist of; 97 field observations, including observation of 25 whole feedback-loops , 28 semi-structured one-to-one interviews with students and approxima-tely 400 web-based e-logs written by the students.
Preliminary analysis seems to give strengths to our assumption that for teacher feedback to be used by these undergraduate students it has to be well-styled and its form needs to be adapted to the type of learning activities (remembering, understanding, applying, analysing, evaluating
11h-12h30: Breakout room 1 Sterling
Feedback matters when students are considering what type of learn-ing activities to engage in - exploring the relationship between feed-back-types and students learning activities in higher education.
Paper
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
61
Paper Perceptions of the purposes of portfolio assessment in primaryeducation.
Diana Baas • Mien Segers • Jos Castelijns • Marjan Vermeulen • Rob MartensNowadays, portfolios are becoming increasingly popular in primary education as a tool to pro-
mote learning. In spite of the popularity of portfolios, different definitions are used and a wide array of purposes is being served, ranging from summative to formative. In addition to various purposes of portfolio assessment, different stakeholders are involved in the portfolio process in primary education. However, when stakeholders have differing perceptions of the purpose of the portfolio, the power of the portfolio as a learning tool can become compromised. This study fo-cuses on stakeholders’ perceptions of the purposes of portfolio assessment in primary education. Furthermore is being studied whether perceptions of the purposes of stakeholders differ. Semi-structured interviews with principals, teachers, students and parents were conducted in ten pri-mary schools. Validated summaries of the interviews were coded with MAXQDA using a grounded theory approach. Results show that stakeholders perceive the purposes of portfolio assessment in primary education to be primarily formative. Portfolios are perceived as a powerful tool to give students responsibility in their own learning process, to monitor their learning and to reflect on learning. Concerning these purposes, no differences in perceptions of the various stakeholder groups could be found. However, stakeholders reported differing perceptions on the use of port-folio assessment for communication and feedback purposes. With respect to these purposes the-re is still room for improvement.
Paper Primary school teachers’ use of classroom assessment in Dutchmathematics education.
M. Veldhuis • M. van den Heuvel-Panhuizen • J. A. Vermeulen • T. EggenIn this study we investigate the classroom assessment practices of Dutch primary school tea-
chers and their view on the role of assessment in their teaching practice. We consider classroom assessment the process of collecting data by teachers about their students’ learning process in such way that teachers can make informed decisions about how to continue in their teaching. From previous research is known that teachers can use of a wide range of assessment methods and that teachers’ use of assessment methods is related to teachers’ assessment purposes and tea-chers’ beliefs concerning classroom assessment. However, despite the use of formative assessment, summative assessment practices have still been found to dominate educational policies of most governments. As a result teachers may perceive assessment aimed at assessing students’ edu-cational needs as less important, which in turn may result in unbalanced classroom assessment practices. Unclear is what this means for the teachers’ classroom assessment practice in mathema-tics education in Dutch primary schools. This study aims at getting a more clear picture of this, and exploring how several classroom assessment aspects are related on the teacher level as well on
11h-12h30: Breakout room 1 Dollar
and creating) the students are willing to engage in at that point of time in their learning processes.
11h-12h30: Breakout room 1 Dollar
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
62
PaperDynamic cognitive assessment of young adults.
Alex KozulinObservation that students’ independent task performance do not necessarily coincide with their
learning potential or cognitive modifiability prompted Vygotsky to formulate the concept of the Zone of Proximal Development (ZPD). Later on the concept of ZPD became an inspiration for so-called dynamic assessment (DA) procedures that focus on students’ ability to benefit from a short mediated learning experience provided during the assessment. The majority of DA studies aimed at demonstrating that DA helps minority children and children with special needs to reveal their “hidden” potential. Many aspects of DA, however, remained unexplored particularly in application to adult learners with typical development: 1) What is the range of learning potential of adult learners who have the same unassisted performance scores; 2) Does learning potential constitute a general ability, or is it modular, dependent on DA task content and modality; 3) Is there a diffe-rence between learning potential demonstrated during DA and cognitive change in response to a long-term educational intervention. These questions were addressed in two studies with large groups of young adults. The results indicate that subjects with the same static test scores have widely different learning potential; learning potential appears to be modular, dependent on the type of DA tasks; learning potential and cognitive modifiability seem to reflect different abilities.
the school level. Data are collected through an online teacher questionnaire. This questionnaire consisted of a number of questions about relevant teacher background characteristics, teachers’ use of assessment methods and teachers’ beliefs about and views on the role of assessment. After conducting a non-response study, we will first use univariate descriptive statistics to describe the classroom assessment practice and teacher and school characteristics, furthermore correlation analyses will be used to find out how assessment methods, uses, and purposes are related (on the school and teacher level), and finally (multi-level) regression analyses to measure the influence of teachers’ beliefs and characteristics on their use of assessment methods.
Paper How do university teachers assess their students? Understand assess-ment from their own viewpoint.
V. Quesada-Serra • G. Rodríguez Gómez • Mª S. Ibarra Sáiz • E. García JiménezRecent tendencies regarding assessment in higher education such as the ones proposed by
Boud (2010), Carless, Joughin and Mok (2006), Falchikov (2005) or Nicol (2010) highlight the need to implement alternative assessment practices to enhance students learning that take into ac-count aspects like: develop authentic assessment tasks, provide quality feedback and feedforward to students and promote student’s active participation in the assessment process. A survey re-search method was carried out with the aim to provide the opinion of university teachers regar-ding their assessment activity, with regards to the importance, competence and use of thirty-one
11h-12h30: Breakout room 1 Yen
11h-12h30: Breakout room 1 Yen
Linking Multiple Perspectives on Assessment Brussels 28-31 August, 2012
63
Paper Assessment literacy in university students: what is it and how is it developed?
Birgit Den OuterTo perform well students must master the material of their chosen subjects but also need to
develop an expertise to help them negotiate assessments as members of a learning community. In this study, the latter requirement is referred to as ‘assessment literacy’, the research into which is approached holistically with the distinction between formative and summative assessment de-liberately blurred. The paper is a presentation of the findings of a two-year project among un-dergraduate and postgraduate students. Central questions are: how do students develop their assessment literacy? What place do assessments take in students’ perceptions of themselves and how do they use assessment experiences of the past to negotiate assessments of the future? Au-dio diaries are used to track experiences of 20 students from several disciplines at a post-1992 university. In addition, data derived from semi-structured interviews further illuminate aspects of these experiences by situating them in respondents’ broader educational (hi(s)tories. An initial conceptualisation of assessment literacy draws on academic literacies models, and educational narrative and developmental identity literatures. Findings to date point towards two propositions: firstly, students use assessment experiences to construct, reinforce, or challenge their narrated self in the domain of learning through which they develop their assessment literacy. Secondly, a developed assessment literacy is where learning from feedback, either summative or formative, is aligned with a narrated self. The paper will illustrate these two propositions with material from the data and discuss the circumstances in which certain types of assessments supported or undermi-ned the research participants’ learner identity and how this is linked to wider, socially constructed notions on success and failure in the learning environment.
assessment tasks previously selected. The sample comprised 427 university teachers from 18 Spa-nish universities; most of them had over ten years of teaching experience. The results show that teachers gave higher scores to those assessment tasks related to ‘assessment planning and design’ and ‘monitoring of student’s learning’. On the contrary, the most innovative practices such as the items related to promoting the ‘participation of students in assessment’ received the lowest scores of the questionnaire -Primarily regarding the ability to put those assessment tasks into practice and the extent to which they already use some of them in their classes-. In general, university teachers considered most of the tasks as very important, the competence and the use of those assessment tasks are, however, lower. Therefore, we can conclude that university teachers see the importance of another type of assessment rather than the traditional. Higher education institu-tions must take part and promote policies that enhance alternative assessment practices. To do so, it is essential that university teachers feel confident and capable enough to introduce them.
11h-12h30: Breakout room 1 Yen
E A R L I S I G 1 : A S S E S S M E N T A N D E VA LUAT I O N
64EARLI SIG 1ASSESSMENT AND EVALUATION