FLEAT VI - Harvard University - Piet Desmet & Bert Wylin

Bridging the gap between closed and open items

or how to make CALL more intelligent

Piet Desmet & Bert Wylin

Fleat VI Harvard University August 11-15, 2015

1. Item-based learning & testing environments (ILTE): definition

2. CALL, SLA & LT: different views on a “classical” ILTE

3. Beyond the closed & open items in an ILTE

4. Half-closed items

5. Half-open items

6. Supported open items

7. Challenges for ILTEs

8. Conclusion

1. Item-based learning & testing environments (ILTE): definition

1.1. Definition of an item

“A digital item asks the learner to react to a given input, leading to an output that is treated by the system”.

Typically, items are• part of a series (or stand on themselves)• structured (organized),• (minimally) metadated,• reusable,• multimedial,• stored in an item bank

1.2. “Classical” items: closed or openCLOSED OPEN

Learner output

level of freedom limited totally free

# correct answers limited to 1 or a few many

predicatibility answers maximal very limited

Output treatment

correction type automated manual

reliability high

Examples

closed: multiple choice, multiple answer, drag & drop, order, fill gaps, etc.

open: upload text file, audio or video-recording (without correction)

2. CALL, SLA & LT: Different views on “classical” ILTEs

2.1. Within CALL: tutor vs tool

Computer as a tutor (tutorial CALL):ILTEs still crucial today although need for improvement

“Many programs being produced today feature little more than visually stimulating variations on the same gap-filling exercises used 40 years ago”

(Beatty 2003: 11)

vs

Computer as a tool (multimedia, CMC, web 2.0, etc.): ILTEs less important since main focus is on CMC, social media, immersive virtual worlds, etc. allowing for communicative activities and tasks

Tutorial CAL not even on the Hype cycle for education (Gartner, 2013)

2.2. Within SLA: cognitive vs socio-cultural

Different perspectives on SLA:

cognitive perspective: cognitive processing by the learner(noticing, motivation, etc.)

socio-cultural perspective: impact of social environment of the learner (collaboration between learners, scaffolding by interlocutor, etc.)

-> ILTEs are more crucial within a cognitive framework

2.3. Within language teaching: behavioral vs communicative/task-based

° Different methods:grammar-translationdirect methodscommunicative approachtask-based language teaching (TBLT)etc.-> ILTEs are considered to be less crucial in TBLT than before (cf. “drill & kill”)

° Different focus:focus on form vs focus on meaningrule-based vs usage-basedknowledge-oriented vs skills-orientedteacher-centered vs learner-centered-> ILTs are mainly associated with the left focuses

3. Beyond the closed & open items in an ILTE

3.1. Limitations of “classical” closed items

(a) too limited freedom at the level of the learner output

(b) too limited cognitive complexity

(c) limited number of item types

(d) less suited for advanced learners

-> need for more “intelligent” CALL

3.2. Old wine in new bottles…

Till recently only technological innovation

floppy disk (DOS only)

cd-rom (Windows)

website

platforms

CMS LMS learning platform testing platform

SPOC MOOC

3.3. “Our” solution: bridging the gap between closed and open items

= pedagogical innovation

still automated correction with high reliability

BUT:

Learner output: more freedommore correct answersless predictability

www.edumatic.com

http://www.delta-associates.com/what-about-the-old-advice-dont-reinvent-the-wheel-is-it-stupid-or-smart/

4. Half-closed items

CLOSED HALF-CLOSED

Learner output

level of freedom limited more free

# correct answers limited to 1 or a few limited

predicatibility answers maximal maximal

Output treatment

correction type automated automated

reliability high high

Examples

(1) select text(2) dictation

4.1. Definition

4.2. Select text

Learner output: selection of relevant passage in a text

The locus of the points of interest is not given beforehand

-> more freedom at the level of the learner output

Mechanism behind these items:

° mark the keyword(s) in a given text (sentence or paragraph) & link/group these keywords

° define ranges for selection (ranges as such don’t influence the score)

° prepare feedback for correct and wrong keywords

Bert: TE VERVANGEN DOOR VB VOOR TAAL!

4.3. Dictation

Learner output: transcription of a (bookmarked) audio file

Learner doesn’t know what are the possible points of interestLearner can decide not to transcribe certain parts (without impacton the correction mechanism)

-> more freedom at the level of the learner output

Mechanism behind these items:

Approximate string matching

Approximate String Matching @ Edumatic

• Normalization of input (or not)• caps• interpunction• accents

• algorithm based on best match with input

I inform you to XXX the (…) tomorrow (XXX).

• 3 codes: delete, insert, substitute (error)

• Attempts model:attempt – feedback – attempt – (…) – solution model

Approximate String Matching @ Edumatic

• “Brackets” model

[[In the/Every] morning, Mary listens to the radio./Mary listens to the radio [in the/every] morning.]

• not only feedback,also show solutions based on best match with student’s input

showing non matching solutions is an option

Bert: VB van gecorrigeerd dictee toevoegen

5. Half-open items

HALF-CLOSED HALF-OPEN

Learner output

level of freedom more free more free

# correct answers limited to 1 or a few many

predicatibility answers maximal limited(but feasable and progressive build up)

Output treatment

correction type automated automated

reliability high average to high

Examples(1) translate(2) reformulate(3) correct

5.1. Definition

5.2. Translate

xxx = substitute

(…) = insert

(xxx) = delete

5.2. Translate 2.0

Correction on the letter level

BE-ODL 21 maart 2006

5.3. Reformulate/correct

6. Open supported

HALF-OPEN OPEN SUPPORTEDLearner output

level of freedom more free free

# correct answers many many

predicatibility answers limited even more limited

Output treatment

correction type automated automated

reliability average to high average to high

6.1. Definition

6.2. Mechanism

• open question with free learner input

• with due date

• generation of feedback on the basis of:

model answerkeyword matching

• white list (+ score)• and• if• if then

• black list (0 or – score)• negations (and range)

4 functions of supported open item type:

1) Creation of open questionwith model answer, black list, white list, elaborated feedback, etc.

2) Publication of this item fix due date, select student groups, follow-up received

answers, etc.3) Half-automated correction of the answers

correction proposal on the basis of the available infomanual correction of scores and adaptation ofblack list & white list (-> update of automatic scores)

4) Generation of feedback report individualised feedback, fix scores, add personal commentsnotify all users by automatically generated mail

Item Input: create New item

Item Input: create New item

Add original text in “logical units”

(paragraph or

sentences)

Add instruction

Students make translations•Use quick codes to have alternative correct solutions

• Eg. [on passe/on passera/on fera/on effectuera/sera passée/sera prise/l'infirmière glissera]

•Decide about keyphrases•Add scores per keyphrase•Add feedback per keyphrase

• including error specific feedback

Student/candidate response

With or without Correction

button (practice vs. exam)

Student/candidate types

answer

Students make translations•While correcting student input,

• Add more options• Update all existing corrections

constantly

•See the effect of the updates in new student input:

• less and less corrections to make• more and more keyphrases

recognized (both correct and wrong answers)

Item Input: create New itemAdd

translation keywords

and keyphrases

Item Input: create New item

Option: set options for

spell checker

Option: provide model answer(for feedback)

Update translations/scores

Update, add, delete

translations

System asks to apply changes in translations to all students

Final reportingBased on updated

translations and scores

See individual and group

results

Bert: TE VERVANGEN DOOR VB VOOR TAAL!

• Use of supported open exercises in three steps

• Step 1 : try outas a marking and feedbacktool (aid) used by teaching staff-> human verification and improvement of the black & white list is necessary

• Step 2 : learning result of scenario 1 can be used as an exercise with full automatic corrective and elaborated feedback (with human intervention!)

-> human verificationand e-mail feedback

• Step 3 : exam simulation results of scenario 2 can be used as an exercise with full immediate automatic corrective and elaborated feedback (without human intervention!)

•!Supported open exercises are not limited to languages

•Excellent experiences in•Law faculty•Medical faculty

7.1. Adaptivity-> frontend: e.g. adaptive item sequencing

adaptive feedback

7.2. Gamification-> frontend: e.g. Badges & rankings

Collaboration & competition7.3. Flexible delivery mode

-> frontend e.g. Integration in App or digital textbookIntegration in skills oriented learning environment

7.4. Output correction through NLP-> from backend to frontend: e.g. parsing half-open input

7.5. Analysis of tracking & logging data-> from backend to frontend: e.g. reporting

7. Challenges for ILTEs

http://ingvihrannar.com/wp-content/uploads/2014/02/testing_cartoon.jpg

7.1. Adaptivity

4D-model of adaptive instruction

Vandewaetere, Desmet & Clarebout 2011 / Vandewaetere & Clarebout, 2012

Cognition (e.g. prior knowledge)

Affect (e.g. motivation)

Behavior (e.g. need for help)

What elements in the environment to adapt?

Adapt during interaction, between interactions, prior to interaction?

Who’s in control?Learner vs. instructor decides what/when/how to adapt?Or both?

http://www.slideshare.net/piet_desmet/2015-0522-presentatiecalicodesmet-vandewaetere-def

7.2. Gamification

MindSnacks ‘Swell’

Using gameplay mechanics for non-game applications

- Challenges embedded in a compelling story

- Various layers or levels & character upgrades

- Rewards (scores & badges)

- Social interacton & peer motivation through competition

http://www.playwarestudios.com/wp-content/uploads/2013/07/gbl-cartoon.jpg

7.3. Flexible delivery mode

“Classical” delivery mode

Items(in Activities)

from: Horton, William, E-Learning by Design, Wiley, 2011

(a) From a technological point of view

ILTE as a

- smartphone app

- daily small interactive e-mail or sms

- micro-series of items, embedded in a digital textbook

- etc.

More flexibility

(b) From a pedagogical point of view

“Skinning” of item types to be integrated in a skills oriented environment

e.g. multimedia learning environment focusing on audio-visual comprehension

e.g. situational judgment test / inbox exercises

www.franel.eu Nedbox

7.4. Output correction through NLP or statistical methods

NLP ASM

- by definition language dependent- high R&D effort

+ by definition language independent+ lower R&D effort

- unequal availability and quality of existing algorithms and tools- technologies not easily transferable to new tools/environments- slow

+ high availability of existing ASM algorithms+ easily reusable algorithms+ higher speed

+ better granularity (fineness with which input can be analyzed)

- highly depending on teacher’s input (number of correct answers predicted by teacher)

+ language specific intelligent feedback generation by the algorithm (cf. E-Tutor T. Heift)

- no automatic language specific feedback generation

NLP: lemmatisation -> tagging -> parsing (-> semantic analysis?)

Statistical methods: combine advantages of ASM & NLP!

Statistical error detection:training a classifier based on a corpus of corrected utterances with feedback

(cf. PhD Ruben Lagatie)

7.5. Analysis of tracking & logging data

From manually entering data to online massive storageFrom self-reporting data to behavioral dataFrom single measurements to longitudinal measurementsFrom inaccessible to everywhere

From big data to rich data…

Not the data, but the views on the data make it interesting…

For the user: - detailed reporting (from generic to specific!)

- advice on next steps

For the teacher: - reporting at individual and group level

- item analysis

For the user: detailed reporting (from generic to specific reports)

For the user: advise on next steps

For the teacher: reporting at group level

Bert: illustratie invoegen!

For the teacher/content author: item analysis

http://ayende.com/blog/2421/when-does-it-make-sense-to-reinvent-the-wheel

8. Conclusion

CLOSED HALF-CLOSED HALF-OPEN OPEN SUPPORTED

OPEN

Learner output

level of freedom

limited more free more free free totally free

# correct answers

limited to 1 or a few

limited many many many

predicatibility answers

maximal maximal limited very limited

very limited

Output treatment

correction type

automated automated automated automated manual

reliability high high average to high

average to high

More info

Piet Desmet Bert Wylin Piet.Desmet@kuleuven.be Bert.Wylin@kuleuven.be

B. Wylin@televic.com

www.linkedin.com/in/pietdesmet www.linkedin.com/in/bertwylin

@PietDesmet

ITEC www.kuleuven.be/itec