Using Automatic Question Generation to Evaluate Questions
Generated by Children
Using Automatic Question Generation to Evaluate Questions
Generated by Children
Wei Chen, Jack MostowGregory AistProject LISTENSchool of
Computer Sciencewww.cs.cmu.edu/~listen Carnegie Mellon
UniversityCommunication Studies and Applied Linguistics Department
of EnglishIowa State University1or off-task speech?2What are you
wondering about now?
Problem: classify speech
Spoken question?
Four Types of ResponsesI wonder how the cool cat survives in the
cold and snowy placeYes I cant wait till this book is over please
tell me this book is over now1. A complete question2. A partial
question3. Off-task speech4. No response3Im wondering how will
Tony
Add audio not already played3
Generate QuestionswindI wonderIm wondering
howwhyifwhenwhowhathowmakes electricityI wonderwhatlives on
Mars4disfluencyFix fonts; title style doesnt match earlier titles;
need more info to motivate segue: Model self-questioning responses:
Use FSG to generate questions; 2. Add arcs to model disfluency; 3.
Add trigrams to model off-task speech.Use real examples. If cant
mix and match N and VP, fix diagram to reflect reality.4Model
Self-Questioning Responsesand Im likeback back backcant go onnow Im
onof it rightokay lets goalready read thisoh my godReyna come
here
I wonderIm wondering
howwhyifwhenwhowhat5Off-task trigramsShow where source text fits
in: animate? Text instantiating THING, VP; mix and match any N w/
any VP?5QG in Response ClassificationASRGenerate Questions6
Feature vector for utteranceLanguage model
Avoid gratuitous color.Distinguish data from process?Distinguish
decision from data What do Questioning and Off-task boxes
represent?Avoid mixing arrows with different meanings (data flow
vs. decision tree)6Classifying Responses
I WONDER IF WAS? A NATIONAL YELLOWSTONE DONE? EXISTSVMPitch,
intensity, duration, MFCC, % off-task words: 4/9% off-task words
with ASR confident: 2/9 % on-task with ASR uncertain:
0QuestioningOff-task
Most informative acoustic features are Use animation to
associate words with %7EvaluationInvolves questioningNo questioning
involvedFeature vector of utteranceRecall: 0.59Precision:
0.85Recall: 0.83Precision: 0.55vs. 0.55vs. 0.76vs. 0.80vs.
0.50question generation vs. story trigrams250 responses in Reading
Tutor by 34 children ages 7-10
Words correctly recognized: 38%
OOV rate: 19%Add story trigrams results as animation.Move
R&P below boxes (moved up to make room).QG-based model may need
concise self-explanatory name.Mark significant differences if
any.8ConclusionGenerated questions to help detect off-task
speech
Generate questions to model on-task speech
Interpolate with language model of off-task speech9Use
consistent red and blue throughout.Clarify structure of text.Move
main point into title?Missing: so what? Future? ? Take-away?
Lesson(s)? How generalize?9
