Explanation in GILA 2

Stanford -> RPIMcGuinness, Ding

January 15, 2008

Motivation

Improve trust in recommendations from GILA components• Support evaluation

– Understand why GILA makes suggestions – Identify which prior knowledge is (re-)used– Identify which learned knowledge is learned and (re-)used– Summarize usage of external interaction information steps

• Support internal trust and reuse– Identify which component suggested what and why– Identify/ propagate dependencies

Provided Knowledgeruntime inputruntime input

prior input prior input

Learned Knowledge

Final Output

Flow of GILA Knowledge

expert execution trace

ontological knowledge

constraint-violation

pstep list as the final solution

Learning & practice

performance

context knowledge

facts embedded inthe input problem

problem/solution

constraints

conflict priority

GILA KnowledgeKnowledge Category Producer Consumer OWL class

Execution trace Provided Expert MRE, ILRs gilcore:ExecutionTrace

Input ACO problem Provided Blueforce MRE, ILRs gilaco:ProblemACO

Context knowledgeProvided Context

serviceMRE, ILRs Phase II

Ontological knowledge Provided Expert MRE, ILRs aco:*

Constraint Learned CL ILRs gilcore:Constraint

Constraint Violation Learned SC MRE, ILRs gilaco:ViolationStatement

An ACO problem state Learned MRE ILRs gilaco:ProblemACO

Cost of an ACO problem state Learned DTL MRE, ILRs? gilaco:CostStatement

Credit/Blame Assignment Learned SC + MRE ILRs Phase IIConflicts in an ACO problem state

Learned 4D MRE gilcore:Conflict

Order of conflicts Learned ILRs MRE Phase II

Intersection of conflicts Learned 4D MRE gkst:IntersectionDetails

Pseudo expert trace Learned MRE, ILRs ILRs gilcore:ExecutionTrace

Final solution Learned MRE-DM PstepManager gilcore:PstepList

Phase II Objectives

• Show usage of provided knowledge– Expert execution trace– Ontological knowledge

• Show usage of learned knowledge– Constraints– Blame (constraint violation) assignments

• Show (abstracted) problem-solving trace (with dependencies)– problem, learner, solution, …., final solution

Approach

• Ontology Representation– Reference knowledge at appropriate level of granularity– Provide interlingua for learners’ knowledge usage

• Extract some knowledge from KB• Derive new knowledge from existing knowledge

• Computational Components– Run time API for sharing knowledge of explanations– API for extracting, summarizing and visualizing problem-

solving and knowledge usage trace utilizing Inference Web approach

Example: Blame-assignment

#vstmt192 (gilaco:ViolationStatement)•Degree of importance of violation•How severe the violation is•Confidence

#constraint 7(gilcore:Constraint)

#pstep261(gilcore:Pstep)

#problem26(gilaco:ProblemTrySafetyConstraint)

#solution28 (gilaco:SolutionTrySafetyConstraint)

SafetyChecker

MRE-DM #solution23(gilaco:SolutionResolveConflict)

#solution26(gilaco:SolutionResolveConflict)

X-ILRDT-ILR

…

Learned knowledge

Learned knowledge

Learned knowledge

Learned knowledge Provided knowledgeProvided knowledge

Example: Learn and Solve Based on correspondence, santi Nov 30, 2007

Solution 5647 Case 101

PSTEP 8859

PSTEP 8860

PSTEP 8861

ExecutionTrace 1878

generartedfrom

learnedfrom

learnedfrom

learnedfrom

in

in

in

0.9054

confidence

0.9054

confidence

Performance Step 129Performance Step 129 Learning Step 126Learning Step 126

CBL ILRCBL ILR

Has conclusion Has antecedent Has conclusion Has antecedent

match case and solvematch case and solve Learn new caseLearn new caseHas learner

Has learnerUse rule Use rule

Directions• Determine appropriate granularity for representation and

propagation (initially coarse level moving to finer granularity where required)

• Design appropriate primitives for phase II topics – context, credit/blame, orderings, priorities, …

• Focus on dependencies initially (supporting explanations showing usage of prior knowledge, external interaction-gained info, use and re-use of learned knowledge, similarity knowledge, adaptation knowledge…)

• Design GILA-appropriate explanation templates exploiting our explanation interlingua

• Present knowledge provenance summaries (with follow up options)

PSTEP Manager

FujitsuSong

PSTEP Manager in Phase 1

SWebADSWebAD

PSTEP Manager (PMAN)

PSTEP List

PSTEP List

Translate into SWebAD

command

Translate into SWebAD

command

Backup PSTEP information

Backup PSTEP information

ExecutionExecution

Generate execution trace

• Known Issues– PMAN is widely used to check

whether a proposed solution is correct or not, each check takes long time.

– Most failures are due to some errors which can be checked before execution and avoided (e.g. NaN maximum altitude, minimum altitude > maximum altitude, etc.)

– The knowledge may not be discovered from the expert trace (since the expert does not make this type of mistakes), but can be learned from the execution result reported by SWebAD during the execution time.

PSTEP Manager in Phase 1 is a simple execution engine and it hides execution related details from other modules

PSTEP Manager in Phase 2

STBMCSSTBMCSPSTEP

ListPSTEP

List

Translate into STBMCS

command

Translate into STBMCS

command

Backup PSTEP information

Backup PSTEP information

Generate execution trace

Early Error DetectionEarly Error Detection

Execution Request

Execution Request

STBMCS Error

Learner

STBMCS Error

Learner

Constraint knowledge repository

Constraint knowledge repository

PSTEP List Optimization

PSTEP List Optimization

New ModulesNew Modules

Existing ModulesExisting Modules

PSTEP Manager (PMAN)

• STBMCS Error learner learns new knowledge from error report of STBMCS (previously SWebAD)

• Share constraint knowledge with 4DCL

• Use constraint knowledge to detect errors of proposed solution before real execution

Share with 4DCL/R

Error Report

Extra

Files/WWW Toolkit

Proof Markup Language (PML)

CWM (NSF TAMI)

JTP(DAML/NIMD)

SPARK(DARPA CALO)

UIMA(DTO NIMD

Exp Aggregation)

IW Explainer/Abstractor

IWBase

IWBrowser

IWSearch

Trust

Justification

Provenance

N3

KIF

SPARK-L

Text Analytics

IWTrust

provenanceregistration

search enginebased publishing

Expert friendlyVisualization

End-user friendly visualization

Trust computationSemantic Discovery Service

(DAML/SNRC)

OWL-S/BPEL

Framework for explaining question answering tasks by • abstracting, storing, exchanging, • combining, annotating, filtering, segmenting, • comparing, and rendering proofs and proof fragments provided by question answerers.

Inference Web Infrastructure McGuinness, Ding, Pinheiro da Silva, Chang, Fikes, Glass, Zeng