Metacognitive Capabilities for Advice Taking

1USC Information Sciences Institute Jihie Kim

Jihie KimYolanda GilJim Blythe

Intelligent Systems DivisionUSC/Information Sciences Institute

www.isi.edu/ikcap/

Metacognitive Capabilities for Advice Taking


Challenges

Systems must understand how user’s advice/guidance/request fits before accepting it.Self-assessment of competence and confidence


Key Idea

Apply metacognitive skills in human learning [White and Frederickson 98,Harris 1990]

Awareness of its own knowledge and the ability to understand, control and manipulate its own cognitive process


Approach

Explicit representation of meta-level goals to understand, control, and manipulate system’s cognitive process of KA and develop strategies for them.Develop annotations to the dialogue history based on the goals and strategies (episodic memory)Apply our techniques in proactive KA dialogue [Kim & Gil AIEd 2002, Gil & Kim CogSci-2001; Kim& Gil ITS-2001]


Meta-Level Goals

Establish confidence on KB itemsEnsure that KB items are useful and run

efficientlySelf-evaluate successful/unsuccessful procedures procedures that are often used vs rarely used onesprocedures that are efficient/inefficientevaluate other KB items in a similar way

Be aware of the goals set (i.e. understand the purpose)

Achieve goals or Ensure progress towards the goals

Achieve proficiencygoals (intentions) that are often seterrors/gaps foundprogress towards a goal (history)answer changes (history)redundant calls to a procedure


Meta-Level Goals (cont)

Ensure that advices were useful in making the problem solving successful

Ask for more advices when there are conflicts, changes in answers, redundancies, etc.

Ensure that advices are consistent with existing knowledgeadvices that were useful for making PS successfuladvices that didn’t affect the resultsfrequent rejects or dislikescontradicting advicesredundant advices

Establish competence: assess own problem solving capabilitykinds of problems that can be solvedkinds of problems that can’t be solved yet (based on attempts)


Developing Strategies for Meta-Level Goals

Goals on KB itemsEstablish confidence on KB itemsEnsure that KB items are useful and run efficiently

self evaluate successful/unsuccessful procedures check the involved PS episodes, whether any of them that

use the procedure succeeded Warning : find_laptop still doesn’t return any results

procedures that are often used vs rarely used ones check how used, how many times Warning: the procedure of ‘find_midrange_systems’ hasn’t

been used for a while.procedures that are efficient/inefficient

check how used, how long it took to run Warning: calls to find_all_laptops took more than 5 min.

evaluate other KB items in a similar way


Developing Strategies for Meta-Level Goals (cont)

Goals on Problem Solving GoalsBe aware of the goals set (i.e. understand the purpose)Achieve goals or Ensure progress towards the goalsAchieve proficiency

monitor goals that are often set check the goals set in the history

goals set but not visited for a while check the goals & PS episodesWarning: are you still interested in send_email to Susie?

errors/gaps foundWarning (if error message not sent from other modules): there are still some

errors … progress towards a goal (history)

number of errors/ number of steps taken in PS episodeFeedback Q: now we can perform find-laptops, but still don’t know how to do

get_autorization answer changes (history)

number of errors/ number of steps taken in PS episodeFeedback: now we have IBM think pad as well as SONY VAIO as the candidates

redundant calls to a procedure Warning: in solving ‘buy laptop’ the same calls to find-laptop have been made

multiple times



Goals on advices takenEnsure that advices were useful in making the problem solving successfulAsk for more advices when there are conflicts, changes in answers,

redundancies, etc.Ensure that advices are consistent with existing knowledge

advices that were useful for making PS successful check changes in the PS episodesFeedback: since you raised the max price, now we can find candidate

laptopsadvices that didn’t affect the results

check changes in the PS episodesWarning: you advice to reduce the min speed didn’t change the answer

frequent rejects or dislikescontradicting advices check inconsistent results redundant advices



Goals on Overall problem solving capabilityEstablish competence: assess own problem solving

capability

kinds of problems that can be solved monitor PSs completed and how well they are

donekinds of problems that can’t be solved yet (based on attempts)

check the answersWarning: we can’t solve find_autorization yet


Summary:Metacognitive Capabilities for Advice Taking

Episodic Memory with Annotations Meta reasoning

- Self assessment of competence- Strategies for resolving conflicts- Relevance (usage) and gaps

KB-1

KB-2

KB changesKB item usage

3/28/03 Jihie buys laptop- Initial task goals- solution proposed- advice: new cost limit- 2nd solution- . . .

9/1/03 Jim buys laptop- . . .

10/28/03 Jihie finds laptop

UserAdvice

Problem Solving

KB

. . .

Advice

Follow-updialogue

KB annotations

Documents

Metacognitive Capabilities for Advice Taking