Speech Analysing Componentin

Automatic Tutoring Systems

Presentation by

Doris Diedrichand

Benjamin Kempe

Survey A Short Introduction to LSA (Latent Semantic Analysis)

What LSA does How effective is LSA How is an LSA representation created Some Caveats

Auto Tutor Course in computer literacy at the U. of Memphis Interface Simulating human tutoring use of LSA working system

Why2 Why Why2 ? Evaluation: KCDs Statistic and Symbolic Representations

LSA (Latent Semantic Analysis)

" Motivation:An automatic tutors „understanding“ of what a student says is limited=> reduction on topics/systems where multiple choice or other simple selections make sense.

" LSA is used to analyse students natural language answers:

" what topics are hit" how good is the answer

In Detail

LSA is used to analyse texts on a topic, to create a statistical representation of keywords with word occurrences (=text).

When a student types in a text, LSA computes a representation of that students text (students answer) which can be compared to the reference text's representation.

So natural language answers are evaluated and if needed, it can be found out which topics are lacking in the answer text.

In an example on http://lsa.colorado.edu texts are presented. Students are asked to write summaries (essaies).

Essay 1:A list of keywords,no relation between them

Essay 2:A copy of the original text

Essay 3:A perfect essay about a completely different topic

Essay 4:A good essay about the topic, containing the right keywords

Good essay

Diffrent topic

Copy of text

Keywords only

0 0,5 1 1,5

Evaluation

LSA: How does it work?The text is represented as a matrix in which each row

stands for a unique word and each column stands for a text passage or other context. Each cell contains the frequency with which the word of its row appears in the passage denoted by its column

Next, the cell entries are subjected to a preliminary transformation, in which each cell frequency is weighted by a function that expresses both the word's importance in the particular passage and the degree to which the word type carries information in the domain of discourse in general

Next, SVD (Single Value Decomposition) is applied to the matrix. This reduces dimensionality.

LSA Caveats

" LSA is a pure statistical representation of words and their common appearance

" LSA lacks information such as that expressed by synatx and used in logic

" LSA mimics lexical and semantical knowledge but has none

" LSA can only capture a limited range of topics at a time: to many topics in one representation lead to a less detailed analysis of students texts

An Introduction to Auto Tutor

A Tutoring systemwhich uses LSA as its Analysing Component

Benjamin Kempe

Why2Another intelligent tutoring system

which uses LSA

The goal of Why2 is to coach students through the process of constructing explanations that are complete and do not contain

any misconceptions.

To do so, Wy2 uses KCDs (Knowledge Construction Dialogues).Those KCDs are eather interactive directed lines of reasoning.

Either to elicit a specific idea, or to remediate a specific misconception.

It is important to select exactly the appropriate KCDs both to give students the KCDs that they do need, and to avoid giving the extraneous KCDs that they do not need.

Why2 uses a measurement system, which is able to evaluate (and so to better) the quality of KCD selection in terms of

KCD Precision: KCDs correctly given / total KCDs given

KCD Recall: KCDs correctly given / KCDs needed

KCD false alarm rate: KCDs incorrectly given / KCDs not needed

KCD precision and recall vary with essay quality: the better a students essay, the more difficult it gets to find good KCDs.

To select appropriate KCDs, Why2 uses a special system to select the dialogues: students answers are evaluated with a statistical system (Rinbow, a Bayes classifier) as well as with a symbolic system (CARMEL et al.).

Additional Symbolic Analysis gives better results(Better selected KCDs: more precision, less false alarms, better recall rate)

Statistical Analysis Symbolic Analysis Combined Analysis0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

Precision

Recall

False alarm

Object of research: Combination of diffrent techniques for good analysis of students essaies.

„Bag of words“ i.e. statistical approaches are fast and low cost but can be tricked.Knowledge based approaches are slow and brittle but more precise and capture nuances.

Natural langage -> CARMEL: symbolic sentence level languageunderstanding->set of first oder logical forms

Natural Language->RAINBOW: naive Bayes classifier.Assigns sentences to classes that are associated with set of logical forms

Logical forms -> DLU: discoures language understanding-> proof trees

Proof treef -> which points are missing, which misconceptions

Natural Language

CARMELsymbolic sentence

level language understanding

RAINBOWnaive Bayes

classifier

first oder logicformulas

first oder logicformulas

DLUDiscourse language

understanding

Proof Tree

Literature" An Introduction to Latent Semantic Analysis.

Landauer, T.K., Folz, P.W., Laham, D. (1998).Discourse Processes, 25, 259-284.

" Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor.Graesser, A.C., Wiemer-Hastings, P., Wiemer-Hastings, K. Harter, D.

" Improving an intelligent tutor's comprehension of students with Latent Semantic Analysis.Wiener-Hastings, P., Wiemer-Hastings, K., Graesser, A.C.

" A Hybrid Language Understanding Approach for Robust Selection of Tutoring Goals.Rosé, C.P., Bhembe, D., Roque, A., Siler, S., Srivastava, R., VanLehn, K.

" http://www.autotutor.de

" http://lsa.colorado.edu

Thank you!

Questions?