Proceedings of NAACL-HLT 2019, pages 114–121Minneapolis, Minnesota, June 2 - June 7, 2019. c©2019 Association for Computational Linguistics

114

Development and Deployment of a Large-Scale Dialog-based IntelligentTutoring System

Shazia Afzal1, Tejas Indulal Dhamecha1, Nirmal Mukhi2, Renuka Sindhgatta3∗,Smit Marvaniya1, Matthew Ventura4, Jessica Yarbro4

1IBM Research - India, 2IBM Research - Yorktown Heights, NY, USA3Queensland University of Technology, Brisbane, Australia, 4Pearson Education, USA

{shaafzal, tidhamecha, smarvani}@in.ibm.com, nmukhi@us.ibm.comrenuka.sr@qut.edu.au, {matthew.ventura, jessica.yarbro}@pearson.com

Abstract

There are significant challenges involved inthe design and implementation of a dialog-based tutoring system (DBT) ranging from do-main engineering to natural language classi-fication and eventually instantiating an adap-tive, personalized dialog strategy. These is-sues are magnified when implementing sucha system at scale and across domains. In thispaper, we describe and reflect on the design,methods, decisions and assessments that ledto the successful deployment of our AI drivenDBT currently being used by several hundredsof college level students for practice and self-regulated study in diverse subjects like Sociol-ogy, Communications, and American Govern-ment.

1 Introduction

Intelligent Tutoring Systems (ITS) have been oneof the ambitions of researchers working in thefields of computer-aided education, learning sci-ences, and, to an extent, computational linguis-tics (Brusilovsky et al., 1996; Graesser et al.,1999; Evens et al., 1997). A special case of ITSis Dialog-based Tutoring (DBT) which is basedon the Socratic principle of cooperative dialoguemeant to stimulate critical thinking and deepercomprehension (Carbonell, 1970; Graesser et al.,1999, 2005).

Dialog-based intelligent tutoring systems(DBT) capture the effectiveness of expert humanteacher-learner interactions by using natural lan-guage dialogue. Since articulation of a responsein natural language involves recall and reflec-tion of relevant knowledge it facilitates deepercomprehension of content. DBT consists of asequence of mixed initiative dialogue moves innatural language to steer learners through varyinglevels of content granularity. A conversation is

∗Work done while at IBM Research - India.

Figure 1: Screenshot of the Watson dialog-based tutor.Notice the transition from a broad question to focusedquestion, intent classification of I am not too sure asneed for help, and answer evaluation.

triggered when the tutor poses a question whichtypically leads to a series of dialog turns directedtowards finer reasoning on relevant concepts.The goal is to scaffold knowledge and provideconstructive remediation akin to expert one-onehuman tutoring. A well known example of DBTis AutoTutor (Graesser et al., 1999, 2005) whileother notable systems include Why2 (VanLehnet al., 2002), CIRCSIM-Tutor (Evens et al., 1997),GuruTutor (Olney et al., 2012), DeepTutor (Ruset al., 2013), and the GIFT framework (Sottilareet al., 2012).

115

Building a DBT is a challenging task as it in-volves balancing of conversational efficiency withthe tutoring goal of personalized adaptive mentor-ing and knowledge assessment. In spite of sus-tained research and development efforts there aremajor challenges that prohibit their vast adoptionin educational practice. Some crucial challengescan be identified as:

1.1 Student Response AnalysisNatural language classification is a critical compo-nent of any DBT and is the very basis for drivingan effective conversation. Performing a context-based interpretation of a student utterance is evenmore challenging owing to the diversity of hu-man language, differences in vocabulary and nu-ances. Consequently, current NLP and AI tech-niques have limitations when applied in an open-response scenario like interaction with a DBT. Asmachine learning techniques rely on good train-ing data to deliver good performance, the consid-erable subjectivity inherent in training-data anno-tation, benchmarking, and reaction to misclassifi-cation errors further impacts the classification ac-curacy.

1.2 Content Design and CreationExtensive content authoring is required to driveany efficient DBT. The content needs to be struc-tured in a way that drives the tutoring agenda whileensuring that knowledge elements render naturallyin a conversational flow. This requires a tremen-dous amount of manual and semi-automated effortfrom subject matter experts. Content schema isalso closely tied to the nature of a domain as dif-ferent subjects would have their respective chal-lenges. For example, creating content for a factualsubject like Maths or Physics is substantially dif-ferent to a subject like Psychology.

1.3 Dialog StrategyDevising a meaningful conversational strategy is anon-trivial aspect as it directly impacts learner en-gagement and therefore learning outcomes. Thereshould be sufficient flexibility and variation in re-sponses to cater dynamically to the state and re-quirements of individual students. Additionally,there should be scope to implement interventionsand scaffolds in order to keep the learner moti-vated. For example, surfacing relevant examplesfrom a textbook when a student struggles or dis-plays lack of knowledge during interaction.

1.4 Evaluation

The overall efficacy of a learning system is pri-marily determined by the learning gains achieved.However, the overall learning experience is signif-icantly impacted by other dimensions such as clas-sification accuracy, response time, style of feed-back, variation in language, robustness and usabil-ity issues. Evaluation should therefore involve aholistic assessment of all factors that eventuallylead to improved learning outcomes.

1.5 Scalability and robustness

The underlying algorithm design is impacted byboth the domain engineering effort involved inscaling the tutor across titles as well as the abil-ity to handle several concurrent users. Existingarchitectures are more often monolithic, therebylimiting the scope of improvements and scale.

In the context of the above challenges we nowdescribe the development and refinement of ourWatson dialog-based tutoring system that has sofar been used by over 2,000 college students inthe domains of Sociology, Communications, andAmerican Government. Development of this sys-tem involved creating AI modules for languageunderstanding, designing system architecture formodularity and scalability, and a significant effortto update various designs and algorithms to incor-porate student feedback received at various mile-stones. Our DBT differs from existing systems interms of domain and load scalability. We approachthe issue of scaling across domains using a semi-automated pipeline for authoring, validation andimprovisation of content. Scaling to load is en-abled by designing tutor modules as cloud-basedREST micro-services.

In the following sections we describe the archi-tecture and iterative refinement of our2 tutor fol-lowed by some qualitative and quantitative evalu-ations from field experiments.

2 Design and Architecture

Our Watson dialog-based tutor provides remedi-ation to learners through natural language dis-course. Systematic turn-taking engages learnersin a conversation style assessment of their mas-tery on domain knowledge. The tutor tracks thelearners progress during the course of interactionand launches appropriate interventions accordingto pre-defined dialog strategies. The main com-ponents of the tutor are shown in Figure 2. The

116

Figure 2: Simplified architecture of the tutor system.

most significant module of the tutor is the NaturalLanguage Response Classifier comprising of twoprimary sub-components: the Intent Classifier andthe Student Response Analyzer (SRA).

The Intent Classifier maps a student utteranceto one of about 25 possible intent classes. The twomain intent classes are: 1) a valid on-topicanswer and 2) a valid question. Otherintent classes include requests for help, requestsfor a hint, an expression of boredom, an insult, agreeting, and so on. This level of response clas-sification is crucial to the effective working ofthe tutor as an error at this stage can have cas-cading effects on the entire dialog flow. Thismodule is designed as a hierarchical classifier,going from broad intents to finer intents. Thevalid on-topic answer category requiresdomain/subject specific training whereas manymeta-cognitive intents can be classified in domainagnostic manner.

Student Response Analysis (SRA) is the taskof labeling student answers with categories thatcan help a dialog system to generate appropriateand effective feedback on errors. It is modeledas a classifier with underlying techniques similarto Textual Entailment (Dzikovska et al.), Seman-tic Textual Similarity (Agirre et al.), and ShortAnswer Grading (Mohler and Mihalcea, 2009).It takes the valid student answer and evaluates itagainst the model reference answer into one of 3categories: correct, partially correct,and incorrect (Saha et al., 2018; Marvaniyaet al., 2018). The SRA in our DBT thus makes useof state of art machine learning techniques to per-form classification with macro-average F1 within7% of that of human agreements; newer models(not reported here) have yielded results within 5%of human agreements. It uses an ensemble of se-mantic and syntactic features obtained from the tu-ple comprising of the question, student answer andthe reference answer. This design makes it suitable

for unseen-questions and, potentially for unseen-domains too. For a new domain or textbook, ifstudent answers for training are not available, theexisting base model can be used in unseen-domainsetting. However, if the training data is available,the classification module can utilize it to improvethe grading performance. In addition to responseclassification, SRA also performs a gap analysison the student answer against the expected modelanswer to generate fill-in-the-blank (FITB) styleprompts dynamically. The output of the NaturalLanguage Response Classifier is used to drive thetutor strategy by continuous evaluation against thedomain model and estimates of mastery from thelearner model. The core of the tutoring frame-work is the domain model which is constructed bycontent experts as a hierarchy of learning objec-tives (LOs). Each LO is further structured into asequence of assertions along with correspondingHints and common misconceptions, if any. Thisformulation of content is designed to elicit knowl-edge gradually and allow fine-grained evaluation.With this domain model design schema, the broadtask of evaluating student’s understanding is bro-ken down into set of more focused short answerevaluation.

This domain model design also enables conver-sational dialogue on a topic and micro-adaptationof tutorial strategy while allowing step-based as-sessment of learners mastery. Mastery is repre-sented in the Learner Model which is updated onthe basis of knowledge assessment on students’answers. The learner model also drives macro-adaptation between topics or LOs as defined in thepedagogical model. The pedagogical model for-malizes the dialog response strategy and next-bestaction plan of the tutor using principles of forma-tive feedback, hinting tactics and content cover-age. Currently we use a rule-based dialog strat-egy instantiated for each student based on theircurrent mastery and interaction history. (Shute,

117

Figure 3: Sequence of tasks carried out to deploy the tutor for a new subject or domain

2008; Hume et al., 1996). The interface model isthe front-end that triggers the UI to enable Tutor-Learner interaction in natural language. As ourDBT is conceived to be used as a supplement todigital learning content, it is made strategicallyavailable to students for revision and practice.

3 Deployment and Scalability

All modules of the tutor are functionally iso-lated and are manifested as RESTful microserviceAPIs. This decoupling is important for scalabledesign. The overarching framework of Orchestra-tion connects all the services. It is implementedin OpenWhisk (Mukhi et al., 2017) which followsa serverless computing based cloud-computingexecution model. The front-end communicateswith the Orchestration via broadly abstracted tutorAPIs only. Depending on the functionality, somemodules are domain agnostic (e.g. learner model,next-best-action), whereas others are not. For thedomain dependent modules, a set of microserviceinstances are spawned per domain. In terms ofthe storage facility, NoSQL IBM Cloudant R©, in-memory data structure store Redis, and Cloud Ob-ject Storage SystemTM are employed. Relativelysmall and structured data (e.g. domain content)is maintained in Cloudant; high frequency updatedata (e.g. student progress) is cached in Redis;whereas large AI models are stored in Cloud Ob-ject Storage. To cater to computational and trafficload, horizontal (in terms of processor and mem-ory) and vertical (in terms of number of instances)scaling is facilitated by Kubernetes and Nginx APIGateway. Specifically in the context of commer-cial ITS systems, scalability has two aspects: de-velopment & load management.

3.1 Scalable to DevelopThis corresponds to the ability to scale the tutorto new subjects/domains. The overall turnaround

time to make the tutor usable for a new subjectshould be reasonably low. Figure 3 shows keytasks required to have the tutor system support anew subject such as creation of the domain model,retraining/transfer-learning of AI/NLP modules,testing the dialog and AI module performance,configuration tuning, and deploying the system.Some of these, such as domain model creation,evaluating and fine tuning the model are humanlabor intensive, whereas others have been auto-mated. By design, all of these are distributableacross multiple humans and machines which sig-nificantly helps formulate a factory model forpreparing the tutor for a new subject. The train-ing and deployment time for a new subject rangesbetween 6 hours - 72 hours, depending on theamount of refinement needed in the scoring mod-els.

One of the key design elements for develop-ment scalability is re-usability of AI/NLP compo-nents with minimal modifications across varioussubjects. The SRA module uses a base model thatcan be used across all subjects. The SRA withonly the base model yields a modest accuracy; it isextended with domain specific unstructured (text)and structured (student answers) to transfer it tothe subject/domain. This facilitates bootstrappingthe module and solves the problem of cold startoften encountered in industry AI product develop-ment.

3.2 Scalable to Load

This refers to the runtime load tolerance of the tu-tor including simultaneous access to possibly sev-eral hundred students. Moreover, student enroll-ments may vastly differ per subjects and studentactivity may be very high during certain periods(e.g. prior to tests), and limited at other times(e.g. breaks). All these aspects require dynamicallocation of resources to various tutor modules.

118

Learning Affordance Items

n=37

Learning Affordance Items

n=37

Dialogue Quality Items

n=37

Response Classification Items

n=37

The tutor recognized when my answers were correct

The tutor helped me learn the material faster than reading alone

Strongly Disagree Disagree Neither Agree Nor Disagree Agree Strongly Agree

Figure 4: Response classification related survey ques-tion results illustrating positive impacts of tutor.

Some number crunching heavy modules benefitmore from vertical scaling by virtue of inherentparallelization whereas gating/orchestrating mod-ules typically benefit more from horizontal scal-ing. Our tutor can support 600 concurrent users(corresponding to∼ 300K students using the tutorregularly over a semester) with error rate < 0.5%and response time < 3s on average. The errorrates and response times are much better for moretypical loads (5 – 50 concurrent students) observedin current deployments.

4 Iterative Improvement of the System:Development Life Cycle

Various micro-experiments were done after ev-ery few internal releases to receive feedback frompotential end users (sub-sampled student popula-tion). These micro-experiments helped evaluatethe product features and robustness. Following aresome of the key improvisations.

4.1 Evolution of the Response Classifier

Originally, the response classification was binary:correct or incorrect. A misclassificationtherefore resulted in either severe penalty or le-niency. To mitigate this risk, the module wasretrained to a three-way classifier, with the low-risk third class as partially correct. Do-ing so, indeed reduced the chance of correct-to-incorrect and vice-versa misclassification.However, in later trials we observed that if a stu-dent’s answer is misclassified, it is better to sug-gest a finer follow up activity, than just provid-ing partial grade. For example, by performing agap analysis on the students answer against theexpected one we dynamically generate fill-in-the-blank (FITB) style prompts or use encouragementpumps based on mastery.

4.2 Content Revision

During trials, we encountered cases, where stu-dent answers were correct in context of the text-

book, but were wrong as per the authored con-tent. This called for a need to update/enrich theauthored content based on the actual student an-swers. As a solution, the students were given theoption to rate the tutor’s response using a thumbsup/down sign. These ratings are tracked to drawan SME (subject matter expert)’s attention to im-prove/rectify the content. This semi-automatedapproach allows for continuous user feedback andincremental training of the underlying modules.

4.3 Improvising the Conversation Tone

Feedback from trials was also leveraged to finetune the tone of the tutor responses to make itmore motivating, less pedantic, personable andmore transparent. For example, if a studentanswer was classified as incorrect, the promptsThat’s incorrect!, That seems tobe a wrong answer and That does notmatch with what I have convey the samemessage but with different levels of critiqueand encouragement. Similarly, variation in thefeedback language was personalized accordingto students ability level and punctuation wasstrategically used to give a sense of enthusiasmand positivity. The overall persona and thepersonalized behavior were carefully revised andtuned according to situational needs. See (Afzalet al., 2019) for more details on the design andevolution of the tutor personality.

4.4 Dialog Flow Refinement

The dialog flow is essentially the entire orches-tration of a tutoring goal, personalized for eachstudent depending on their progress and masteryon specific content. Although, our initial dialogflow included two main activities of student an-swering questions and asking questions, we addedadditional activities to the dialog to adapt it to stu-dent mastery and improve engagement. These in-cluded recommending question asking based onquestions asked by peers, presenting simpler bi-nary true/false questions or presenting a fill-in-theblank question if the computed hint question dif-ficulty is high and student mastery is low for thetopic.

5 Evaluation

As of August 2018, the Watson dialog-based tutoris available with 3 titles commercially, with a num-ber of additional titles in development (Ventura

119

Human-2C P I

Human-1C 57.6 20.2 22.2P 5.8 27.2 67.0I 3.1 5.3 91.6

Macro-F1: 60.2%

TutorC P I

Human-1C 64.8 19.5 15.7P 31.9 23.4 44.7I 5.7 20.8 73.5

Macro-F1: 53.2%

TutorC P I

Human-2C 69.0 17.4 13.6P 32.0 34.4 33.6I 11.4 13.8 74.8

Macro-F1: 55.9%

Table 1: Confusion matrices (values in percent-age) of Human-1 vs Human-2 and Humans vs Tutorfor response classification during a learning experi-ment. C, P, and I represent correct, partiallycorrect, and incorrect grade.

et al., 2018). During the fall 2018 semester, thecommercially available titles were used by over2000 students across more than 200 higher edu-cation institutions (Pearson, 2018). The system isexpected to be used across a multitude of highereducation institutions in the USA and will be pi-loted on five additional titles.

Here, we present our initial results gatheredfrom two controlled learning experiments (LE)conducted 7 months apart. The first LE was con-ducted with 39 students while the second LE had102 students. The demographic profile was sim-ilar with average GPA of ∼ 3.5 and 70% femaleratio. Here we discuss some relevant results onthe following key dimensions:

5.1 Classification

For our two-level classification system - at intentand answer assessment level (SRA), we observedthat intent classification of student utterances is95% accurate, and the SRA system for scoring an-swers is within 7% of human inter-rater reliabil-ity for 3 of the 4 learning objectives tested. Fig-ure 4 shows the student responses to survey ques-tions that measure the efficacy of the tutor whileTable 1 illustrates the evaluation of classificationthrough formal metrics. The confusion matricesare reported for 50 transcripts pertaining to to-tal of 1,065 student responses. Note that, I-to-Cmisclassification is 2.6-8.3% higher than that ofhuman disagreement; whereas C-to-I misclassi-fication rate is better than human disagreement.

User’s feedback on how well the tutor understoodtheir responses showed a statistically significantacross the two LEs.

5.2 Validity

The tutor’s estimate of student mastery scores wasfound to correlate significantly with the student’sself-perception (r = 0.32, p < 0.05) and ac-tual post–test learning measures (r = 0.53, p <0.001). This is a strong indicator of the validityof the tutor’s mastery measurement. Note that thisvalidity is crucial as it is the basis for adjustingthe dialog strategy to enable true personalizationof the tutoring experience.

5.3 Dialog Quality

This was estimated by surveying the SMEs andstudents who used the system. The SMEs manu-ally scored 236 conversational transcripts and tooka survey after each transcript. In these surveys, thetutor was rated as providing an effective tutoringconversation 70% of the time, transitioning effec-tively through the conversation 75% of the timeand providing appropriate feedback 66% of thetime. Students’ self–report was also fairly posi-tive about the overall experience, feedback, valueand usability of the tutor.

5.4 Iterative Improvement

Figure 5 reports the improvements observedacross the two LEs on some survey items. Theseimprovements are attributed to content refinement,response and intent classification improvement,and dialog refinement.

Perhaps more gratifying than the generally pos-itive results are the expressions of enthusiasm, in-terest and engagement from students working withthe Watson dialog-based tutor. Comments suchas “I wasn’t having to go over reading materialand lecture material over and over to understandit more fully because I grasped it the first time. Icould take that time to devote to another subject,or my family” was evidence of the truly beneficialsocietal impact that such a system could have, ifapplied the right way.

6 Discussion: Human Subjectivity

We observed that the subjectivity inherent in hu-man language impacts the content authoring, cre-ation of training data and, eventually, the end userexperience. The domain model authoring raises

120

1 2 3 4 5

The tutor gave additional information that helped me learn thematerial

The Tutor helped me learn the material faster than just reading

The Tutor did a good job guiding me to learning the concept

The tutor helped me understand the material more than readingalone

The tutor helped me think more deeply about the material

The tutor understood what I was saying

LE1 LE2

Figure 5: Comparison of LE1 & LE2 on survey itemsscored on 1-5 scale.

issues pertaining to the granularity of content andthe resulting ambiguity expected in responses. Forexample (See Figure 6a), we observe that recalltype question with WHAT interrogative pronounsare more likely to be answered correctly comparedto HOW questions which may be ambiguous due torelative openness of expected answers. This im-plies that classification accuracy can be improvedif a more standardized vocabulary is followed tolimit the variation in content creation. However,this is a debatable proposition since it undervaluesthe richness of domain knowledge and may not bepractical for non-factual domains.

Additionally, given a domain authored contentlike a QA pair, it is hard even for experts to deter-mine the boundaries of correctness in student re-sponses with respect to a reference model answer.This poor inter-rater agreement directly impactsthe effectiveness of training the underlying scor-ing modules. To give an estimate, in one of ourtraining data sets, three groundtruth labels wereobtained from domain experts for each student an-swer. There was disagreement on 50% of the an-swers marked as partially correct by experts! Al-though this is a classical problem in ML/NLP, theconsequences of scoring a correct answer incor-rectly or vice-versa are more profound in a learn-ing system. Human teachers are more flexible andsometimes abstract in their evaluation of an open-ended answer which is difficult to replicate in anAI system.

Finally, it is the delivery of an overall engag-ing and valuable experience that matters to humanend users. Even here, there is remarkable subjec-tivity in users’ perception and tolerance to the typeand timing of tutor misclassifications (Afzal et al.,2018). We cannot measure efficacy solely in termsof absolute and restrictive metrics like accuracyand macro-average F1. Our best proxies, then, are

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0 0.5 1 1.5 2 2.5 3

NO

RMAL

IZED

FREQ

UEN

CY

AVERAGE SCORE PER QUESTION

How

What

(a)

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

NO

RMAL

IZED

FREQ

UEN

CY

F1

mean:0.56std:0.19

(b)

Figure 6: Quantifying issues related to human subjec-tivity: (a) ambiguity in answer correctness as functionof question type, (b) poor inter-annotator agreement.

qualitative, detailed and timely feedback from theend users to learn and improvise over time.

7 Summary

Conversational tutoring is an important form ofnext-generation personalized adaptive educationaltechnology. In this paper we have describedthe design and iterative development of Watsondialog-based tutor – a large-scale DBT that is op-timized to scale across domain/subjects as wellas usage. Its modules are functionally isolatedto facilitate development and runtime scalability.We have described various challenges related tocontent creation and design including their impacton classification performance, refinement of feed-back phrasing and tone, and dialog strategy. Wehave highlighted issues that arise from the inher-ent diversity of human language and how they im-pact the functioning of the tutor and the generatedlearning experiences. On the design side, automa-tion of content extraction techniques can signifi-cantly speed up the content scaling process and al-low building of richer domain models by makinguse of learning material from additional sources.On the experience side, substantial effort is neededto accurately understand natural language and useit strategically to deliver a naturalistic conversa-tional interface that replicates the effectiveness ofhuman teacher-learner interactions.

121

ReferencesShazia Afzal, Bryan Dempsey, Cassius D’Helon, Nir-

mal Mukhi, Milena Pribic, Aaron Sickler, PeggyStrong, Mira Vanchiswar, and Lorin Wilde. 2019.The personality of ai systems in education: Experi-ences with the watson tutor, a one-on-one virtual tu-toring system. Childhood Education, 95(1):44–52.

Shazia Afzal, Vinay Shashidhar, Renuka Sindhgatta,and Bikram Sengupta. 2018. Impact of tutor errorson student engagement in a dialog based intelligenttutoring system. In ITS, pages 267–273. Springer.

Eneko Agirre, Mona Diab, Daniel Cer, and AitorGonzalez-Agirre. Semeval-2012 task 6: A pilot onsemantic textual similarity. In SemEval-2012, pages385–393. ACL.

Peter Brusilovsky, Elmar Schwarz, and Gerhard Weber.1996. ELM-ART: an intelligent tutoring system onworld wide web. In ITS, pages 261–269.

Jaime R Carbonell. 1970. AI in CAI: An artificial-intelligence approach to computer-assisted instruc-tion. IEEE TMMS, 11(4):190–202.

Myroslava Dzikovska, Rodney Nielsen, Chris Brew,Claudia Leacock, Danilo Giampiccolo, Luisa Ben-tivogli, Peter Clark, Ido Dagan, and Hoa TrangDang. Semeval-2013 task 7: The joint student re-sponse analysis and 8th recognizing textual entail-ment challenge. In SemEval-2013, volume 2, pages263–274. ACL.

Martha W Evens, Ru-Charn Chang, Yoon Hee Lee,Leem Seop Shim, Chong Woo Woo, YuemeiZhang, Joel A Michael, and Allen A Rovick. 1997.Circsim-tutor: An intelligent tutoring system usingnatural language dialogue. In ANLP: Descriptionsof system demonstrations and videos, pages 13–14.ACL.

Arthur C Graesser, Patrick Chipman, Brian C Haynes,and Andrew Olney. 2005. Autotutor: An intelli-gent tutoring system with mixed-initiative dialogue.IEEE ToE, 48(4):612–618.

Arthur C Graesser, Katja Wiemer-Hastings, PeterWiemer-Hastings, Roger Kreuz, Tutoring ResearchGroup, et al. 1999. AutoTutor: a simulation of a hu-man tutor. Cognitive Systems Research, 1(1):35–51.

Gregory Hume, Joel Michael, Allen Rovick, andMartha Evens. 1996. The use of hints by humanand computer tutors: the consequences of the tutor-ing protocol. In ICLS, pages 135–142.

Smit Marvaniya, Swarnadeep Saha, Tejas I Dhamecha,Peter Foltz, Renuka Sindhgatta, and Bikram Sen-gupta. 2018. Creating scoring rubric from repre-sentative student answers for improved short answergrading. In CIKM, pages 993–1002. ACM.

Michael Mohler and Rada Mihalcea. 2009. Text-to-text semantic similarity for automatic short answergrading. In EACL, pages 567–575. ACL.

Nirmal K Mukhi, Srijith Prabhu, and Bruce Slawson.2017. Using a serverless framework for implement-ing a cognitive tutor: experiences and issues. InACM WoSC, pages 11–15.

Andrew M Olney, Natalie K Person, and Arthur CGraesser. 2012. Guru: designing a conversa-tional expert intelligent tutoring system. In Cross-disciplinary advances in applied natural languageprocessing: Issues and approaches, pages 156–171.IGI Global.

Pearson. 2018. Revel with watson. https://www.pearson.com/us/higher-education/products-services-teaching/digital-learning-environments/revel-with-watson.html.

Vasile Rus, Sidney DMello, Xiangen Hu, and ArthurGraesser. 2013. Recent advances in conversationalintelligent tutoring systems. AI magazine, 34(3):42–54.

Swarnadeep Saha, Tejas I Dhamecha, Smit Marvaniya,Renuka Sindhgatta, and Bikram Sengupta. 2018.Sentence level or token level features for automaticshort answer grading?: Use both. In AIED, pages503–517.

Valerie J Shute. 2008. Focus on formative feedback.Sage RER, 78(1):153–189.

Robert A Sottilare, Keith W Brawner, Benjamin SGoldberg, and Heather K Holden. 2012. The gen-eralized intelligent framework for tutoring (GIFT).Orlando, FL: US Army Research Laboratory–Human Research & Engineering Directorate.

Kurt VanLehn, Pamela W Jordan, Carolyn P Rose, Du-misizwe Bhembe, Michael Bottner, Andy Gaydos,Maxim Makatchev, Umarani Pappuswamy, MichaelRingenberg, Antonio Roque, et al. 2002. The ar-chitecture of Why2-Atlas: A coach for qualitativephysics essay writing. In ITS, pages 158–167.

Matthew Ventura, Maria Chang, Peter Foltz, Nir-mal Mukhi, Jessica Yarbro, Anne Pier Salverda,John Behrens, Jae-wook Ahn, Tengfei Ma, Tejas IDhamecha, et al. 2018. Preliminary evaluations ofa dialogue-based digital tutor. In AIED, pages 480–483.