SFB 833 Bedeutungskonstitution
Transferprojekt T1
Developing A Web-based Workbookfor English
Supporting the Interaction of Students and Teachers
Björn Rudzewitz, Ramon Ziai,Kordula De Kuthy & Detmar Meurers
Joint 6th NLP4CALL and 2nd NLP4LA workshop
Gothenburg, Sweden, 22 May, 2017
Introduction Adaptation System Error Annotation Outlook
Introduction
• SLA and FLT research has stressed the importance ofindividualized, immediate feedback on learner production.
• Problem: limited opportunity for individual, immediate feedback
• In the classroom, the teacher is generally the only source ofreliable and accurate feedback available to students.
- no time to focus on individual students- difficult to take heterogeneity of students into account
• Outside of class, how can students be supported in a fair way(not relying on parents)?
2 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Tutoring Sytems
• Intelligent Tutoring Systems (ITS) can help address this issue- interactive and adaptive to individual student- quite a bit of research (cf., e.g. Heift and Schulze 2007)- but virtually absent from real-life formal teaching
→ Our goals:- close gap between ITS research, FLT insights, and real-life classroom- address real formal education needs using current NLP technology
3 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
FeedBook: a web-based workbook for English
• Starting point: Camden Town Gymnasium 3 Worbook- approved for 7th grade English classes in German secondary schools→ existing workbook, already integrated into real-life formal education
• Our FeedBook system provides a web-based implementationof the traditional print workbook enabling
- students to complete activities online- teachers to give formative and summative feedback
• The ultimate goal of the system:- to provide individualized, immediate scaffolding feedback to learners- to guide them towards solutions for a number of different activities.
4 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Adapting a paper-based workbook
• Goal: Improve the learning experience for the students andsupport the teachers with minimal overhead.
- make FeedBook as similar as possible to look and feel of print version⇒ without training, users familiar with print workbook immediately benefit
• Online version adds some functionality:- interaction, navigation, user management, . . .
5 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Paper Workbook: Example Activity
85
Diverse Britain 5B5 Extra: Viewing: Exciting things happening at Brixton Village indoor market
Brixton Village indoor market Viewing, Textbook p. 139
a) Watch the clip and find out why the man is so excited.
The man is excited because a new project has opened
in Brixton Village.
There are lots of new shops and galleries for people
to visit.
b) Watch it again and answer the following questions.
1. When does the report take place?
On a snowy December night
2. What was the situation in the market up to 6 weeks ago?
many/20 empty shops
3. What has happened since?
meeting for people to look at empty shops and come up with ideas;
since then, the 30 people with the best ideas have opened up shops
4. What kind of shops can you nd here now? Name 2.
two of the following: old-fashioned sweet shop, (vintage) clothes shops,
lantern maker, ethical fashion designer, furniture restorer, pop-up galleries,
a community shop
5. What is the man’s message to his viewers?
come to Brixton, check out the project and be part of the future of
Brixton Village
B6 Words: Find the right nouns You can make nouns from all these verbs. Find the right nouns and sort them into the grid.
-ation -ion -ing -y no ending
celebration
combination
expression
impression
ending
meaning
apologychange
challenge
end
report
Working with words, Textbook p. 146
apologize · celebrate · change · challenge · combine · end · express · impress · mean · report6 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
First Step: Activity Selection
Selection of activities based on analysis of tasks in print workbook:
1. activity type (e.g. reading comprehension, fill-in-the-blanks)
2. expected type of system input (e.g. check marks, single characters,words, sentences, short texts)
3. form or meaning orientation4. expected language variation (well-formed and ill-formed) based on
task context5. language forms targeted explicitly or implicitly (e.g. simple past
vs. past perfect)
7 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Activity Types in FeedBook
• Overall goal: link real-life needs to NLP-based solutions→ focus on activities for which learners produce language,
and the language produced can be compared to target answers.
• Two main activity types:- short answers, requiring sentences, usually meaning-oriented- fill-in-the-blanks, usually oriented at lexical content or form
• Generalizing across individual activity instances allows for- straightforward inclusion of new instances, and- application of the same processing strategies for similar activities.
8 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
The FeedBook System
• Platform-independent web application
• System supports common workflow in schools:1. students work on exercises assigned as homework
2. students submit results to their teacher
3. teachers correct student answers and send back corrected exerciseswith feedback
4. students inspect the teacher’s results
9 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
FeedBook - Student Lobby
10 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
An example task: listening comprehension
• Each subtask displayed onone page
• Each page contains allnecessary information andmedia
• Student can save or submitexercise.
11 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
An example task: fill-in-the-blanks
• Each subtask displayed onone page
• Each page contains allnecessary information andmedia
• Student can save or submitexercise.
12 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
FeedBook - Teacher Lobby
• System shows overview of student submissionsand indicates tasks to be corrected.
• Teacher selects exercise for correction.
13 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Correction Interface
• Interface shows:
- complete exercise- student answers- target answers- correction aid
• Task of the teacher:
- mark andcategorize errors
- give optionalcomments andrating
14 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Correction Aid: Visual Highlighting
• Student answer isstring-matched againstprestored target answer.
• If match is positive, studentsanswer is marked as correct
• Diff-like algorithm finds andhighlights the parts ofstudent answer that differfrom target answer.
15 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Error Annotation by Teachers
• Teacher selects part of thestudent answer and
• Chooses error categorydescribing nature of divergence
• Optional:
- Free-text comment forstudent
- Correct solution(automatically provided)
16 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Error types in FeedBook
• based on categories provided by teachers in a pilot study
Language form errors Content errorsgrammar problematic understandingspelling missing informationagreement wrong informationclause structure lack of understandingtense extra informationdeterminer alternate answerpronounprepositionword choicemissing wordword orderpunctuation
17 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Correction Aid: Feedback Memory
• FeedBook rememberspreviously given feedback toa given prompt,
• presents it to the teacher atcorrection time.
• Teacher can change ormodify the error correction.
18 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Result of Teacher Intervention
• Teacher error annotationvisually distinguished
• Incremental increase ofsystem coverage andconsistency
19 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Towards Automatically Diagnosing Specific Errors
• If no previous feedback from teachers is applicable:- Provide automatic suggestions based on NLP techniques- First step: grammatical errors in fill-in-the-blanks activities
• Offline processing of target answers:- State-of-the-art NLP architecture (UIMA, Ferrucci and Lally 2004)
- NLP steps: segmentation, POS tagging, lemmatization,morphological analysis and dependency parsing using DKPro(de Castilho and Gurevych 2014)
- Manual correction of the automatically obtained linguistic information
• When teacher selects Feedback interface:- Online processing of student answers- Rule-based comparison of annotated student and target answers to
classify divergences (e.g. wrong inflection→ tense error)
20 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Result Interface for Students
21 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Diagnostics Interface for Teachers
• Teachers can group and visualize errors e.g. by task→ allows targeting specific problems in classroom
22 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Summing up
• The FeedBook system provides- an opportunity for students to individually practice using online
exercises at any time.
- automatic error annotation assistance and a feedback memory tofacilitate the work of the teacher.
- relieves teachers from the repetitive work of providing feedback on thesame issues over and over again,
- while at the same time allowing them to view aggregates of studentperformance
• First version of FeedBook used in pilot classrooms sinceOctober 2016, positive feedback from teachers and students
23 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Next Steps
• Automate feedback for more types of student input- More complex NLP, especially for short answer tasks & content errors
- Integrate short answer assessment system such as CoMiC(Meurers, Ziai, Ott, and Bailey 2011)
• Provide immediate automatic feedback directly to students- Learn and generalize from gathered teacher feedback data
- Use scaffolding to guide student towards correct answer
• Evaluate impact of FeedBook on learning outcomes- Randomized controlled field study integrating measures of the
process and product of learning
- Compare web-based workbook with automated feedback to a versiononly transmitting teacher feedback to students
24 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
Questions?Contact:
Björn [email protected]
Ramon [email protected]
Kordula De [email protected]
Detmar [email protected]
Website: http://feedbook.website
25 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen
Introduction Adaptation System Error Annotation Outlook
References
Richard Eckart de Castilho and Iryna Gurevych. A broad-coverage collection of portable NLPcomponents for building shareable analysis pipelines. In Proceedings of the Workshop onOpen Infrastructures and Analysis Frameworks for HLT (OIAF4HLT) at COLING 2014, pages1–11, Dublin, Ireland, 2014. ACL and Dublin City University.
David Ferrucci and Adam Lally. UIMA: An architectural approach to unstructured informationprocessing in the corporate research environment. Natural Language Engineering, 10(3–4):327–348, 2004. URL http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=252253&fulltextType=RA&fileId=S1351324904003523.
Trude Heift and Mathias Schulze. Errors and Intelligence in Computer-Assisted LanguageLearning: Parsers and Pedagogues. Routledge, 2007.
Detmar Meurers, Ramon Ziai, Niels Ott, and Stacey Bailey. Integrating parallel analysismodules to evaluate the meaning of answers to reading comprehension questions. IJCEELL.Special Issue on Automatic Free-text Evaluation, 21(4):355–369, 2011. URLhttp://www.inderscience.com/info/inarticle.php?artid=42793.
25 | Gothenburg, May 23, 2017 SFB 833, Universität Tübingen