SFB 833 Bedeutungskonstitution

Transferprojekt T1

Developing A Web-based Workbookfor English

Supporting the Interaction of Students and Teachers

Björn Rudzewitz, Ramon Ziai,Kordula De Kuthy & Detmar Meurers

Joint 6th NLP4CALL and 2nd NLP4LA workshop

Gothenburg, Sweden, 22 May, 2017

Introduction Adaptation System Error Annotation Outlook

Introduction

• SLA and FLT research has stressed the importance ofindividualized, immediate feedback on learner production.

• Problem: limited opportunity for individual, immediate feedback

• In the classroom, the teacher is generally the only source ofreliable and accurate feedback available to students.

- no time to focus on individual students- difficult to take heterogeneity of students into account

• Outside of class, how can students be supported in a fair way(not relying on parents)?

2 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Tutoring Sytems

• Intelligent Tutoring Systems (ITS) can help address this issue- interactive and adaptive to individual student- quite a bit of research (cf., e.g. Heift and Schulze 2007)- but virtually absent from real-life formal teaching

→ Our goals:- close gap between ITS research, FLT insights, and real-life classroom- address real formal education needs using current NLP technology

3 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

FeedBook: a web-based workbook for English

• Starting point: Camden Town Gymnasium 3 Worbook- approved for 7th grade English classes in German secondary schools→ existing workbook, already integrated into real-life formal education

• Our FeedBook system provides a web-based implementationof the traditional print workbook enabling

- students to complete activities online- teachers to give formative and summative feedback

• The ultimate goal of the system:- to provide individualized, immediate scaffolding feedback to learners- to guide them towards solutions for a number of different activities.

4 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Adapting a paper-based workbook

• Goal: Improve the learning experience for the students andsupport the teachers with minimal overhead.

- make FeedBook as similar as possible to look and feel of print version⇒ without training, users familiar with print workbook immediately benefit

• Online version adds some functionality:- interaction, navigation, user management, . . .

5 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Paper Workbook: Example Activity

85

Diverse Britain 5B5 Extra: Viewing: Exciting things happening at Brixton Village indoor market

Brixton Village indoor market Viewing, Textbook p. 139

a) Watch the clip and find out why the man is so excited.

The man is excited because a new project has opened

in Brixton Village.

There are lots of new shops and galleries for people

to visit.

b) Watch it again and answer the following questions.

1. When does the report take place?

On a snowy December night

2. What was the situation in the market up to 6 weeks ago?

many/20 empty shops

3. What has happened since?

meeting for people to look at empty shops and come up with ideas;

since then, the 30 people with the best ideas have opened up shops

4. What kind of shops can you nd here now? Name 2.

two of the following: old-fashioned sweet shop, (vintage) clothes shops,

lantern maker, ethical fashion designer, furniture restorer, pop-up galleries,

a community shop

5. What is the man’s message to his viewers?

come to Brixton, check out the project and be part of the future of

Brixton Village

B6 Words: Find the right nouns You can make nouns from all these verbs. Find the right nouns and sort them into the grid.

-ation -ion -ing -y no ending

celebration

combination

expression

impression

ending

meaning

apologychange

challenge

end

report

Working with words, Textbook p. 146

apologize · celebrate · change · challenge · combine · end · express · impress · mean · report6 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

First Step: Activity Selection

Selection of activities based on analysis of tasks in print workbook:

1. activity type (e.g. reading comprehension, fill-in-the-blanks)

2. expected type of system input (e.g. check marks, single characters,words, sentences, short texts)

3. form or meaning orientation4. expected language variation (well-formed and ill-formed) based on

task context5. language forms targeted explicitly or implicitly (e.g. simple past

vs. past perfect)

7 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Activity Types in FeedBook

• Overall goal: link real-life needs to NLP-based solutions→ focus on activities for which learners produce language,

and the language produced can be compared to target answers.

• Two main activity types:- short answers, requiring sentences, usually meaning-oriented- fill-in-the-blanks, usually oriented at lexical content or form

• Generalizing across individual activity instances allows for- straightforward inclusion of new instances, and- application of the same processing strategies for similar activities.

8 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

The FeedBook System

• Platform-independent web application

• System supports common workflow in schools:1. students work on exercises assigned as homework

2. students submit results to their teacher

3. teachers correct student answers and send back corrected exerciseswith feedback

4. students inspect the teacher’s results

9 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

FeedBook - Student Lobby

10 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

An example task: listening comprehension

• Each subtask displayed onone page

• Each page contains allnecessary information andmedia

• Student can save or submitexercise.

11 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

An example task: fill-in-the-blanks

• Each subtask displayed onone page

• Each page contains allnecessary information andmedia

• Student can save or submitexercise.

12 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

FeedBook - Teacher Lobby

• System shows overview of student submissionsand indicates tasks to be corrected.

• Teacher selects exercise for correction.

13 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Correction Interface

• Interface shows:

- complete exercise- student answers- target answers- correction aid

• Task of the teacher:

- mark andcategorize errors

- give optionalcomments andrating

14 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Correction Aid: Visual Highlighting

• Student answer isstring-matched againstprestored target answer.

• If match is positive, studentsanswer is marked as correct

• Diff-like algorithm finds andhighlights the parts ofstudent answer that differfrom target answer.

15 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Error Annotation by Teachers

• Teacher selects part of thestudent answer and

• Chooses error categorydescribing nature of divergence

• Optional:

- Free-text comment forstudent

- Correct solution(automatically provided)

16 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Error types in FeedBook

• based on categories provided by teachers in a pilot study

Language form errors Content errorsgrammar problematic understandingspelling missing informationagreement wrong informationclause structure lack of understandingtense extra informationdeterminer alternate answerpronounprepositionword choicemissing wordword orderpunctuation

17 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Correction Aid: Feedback Memory

• FeedBook rememberspreviously given feedback toa given prompt,

• presents it to the teacher atcorrection time.

• Teacher can change ormodify the error correction.

18 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Result of Teacher Intervention

• Teacher error annotationvisually distinguished

• Incremental increase ofsystem coverage andconsistency

19 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Towards Automatically Diagnosing Specific Errors

• If no previous feedback from teachers is applicable:- Provide automatic suggestions based on NLP techniques- First step: grammatical errors in fill-in-the-blanks activities

• Offline processing of target answers:- State-of-the-art NLP architecture (UIMA, Ferrucci and Lally 2004)

- NLP steps: segmentation, POS tagging, lemmatization,morphological analysis and dependency parsing using DKPro(de Castilho and Gurevych 2014)

- Manual correction of the automatically obtained linguistic information

• When teacher selects Feedback interface:- Online processing of student answers- Rule-based comparison of annotated student and target answers to

classify divergences (e.g. wrong inflection→ tense error)

20 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Result Interface for Students

21 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Diagnostics Interface for Teachers

• Teachers can group and visualize errors e.g. by task→ allows targeting specific problems in classroom

22 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Summing up

• The FeedBook system provides- an opportunity for students to individually practice using online

exercises at any time.

- automatic error annotation assistance and a feedback memory tofacilitate the work of the teacher.

- relieves teachers from the repetitive work of providing feedback on thesame issues over and over again,

- while at the same time allowing them to view aggregates of studentperformance

• First version of FeedBook used in pilot classrooms sinceOctober 2016, positive feedback from teachers and students

23 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Next Steps

• Automate feedback for more types of student input- More complex NLP, especially for short answer tasks & content errors

- Integrate short answer assessment system such as CoMiC(Meurers, Ziai, Ott, and Bailey 2011)

• Provide immediate automatic feedback directly to students- Learn and generalize from gathered teacher feedback data

- Use scaffolding to guide student towards correct answer

• Evaluate impact of FeedBook on learning outcomes- Randomized controlled field study integrating measures of the

process and product of learning

- Compare web-based workbook with automated feedback to a versiononly transmitting teacher feedback to students

24 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

Questions?Contact:

Björn Rudzewitzbjoern.rudzewitz@uni-tuebingen.de

Ramon Ziairamon.ziai@uni-tuebingen.de

Kordula De Kuthyde-kuthy@uni-tuebingen.de

Detmar Meurersdetmar.meurers@uni-tuebingen.de

Website: http://feedbook.website

25 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen

Introduction Adaptation System Error Annotation Outlook

References

Richard Eckart de Castilho and Iryna Gurevych. A broad-coverage collection of portable NLPcomponents for building shareable analysis pipelines. In Proceedings of the Workshop onOpen Infrastructures and Analysis Frameworks for HLT (OIAF4HLT) at COLING 2014, pages1–11, Dublin, Ireland, 2014. ACL and Dublin City University.

David Ferrucci and Adam Lally. UIMA: An architectural approach to unstructured informationprocessing in the corporate research environment. Natural Language Engineering, 10(3–4):327–348, 2004. URL http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=252253&fulltextType=RA&fileId=S1351324904003523.

Trude Heift and Mathias Schulze. Errors and Intelligence in Computer-Assisted LanguageLearning: Parsers and Pedagogues. Routledge, 2007.

Detmar Meurers, Ramon Ziai, Niels Ott, and Stacey Bailey. Integrating parallel analysismodules to evaluate the meaning of answers to reading comprehension questions. IJCEELL.Special Issue on Automatic Free-text Evaluation, 21(4):355–369, 2011. URLhttp://www.inderscience.com/info/inarticle.php?artid=42793.

25 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen