Upload
miriam-gallagher
View
29
Download
1
Embed Size (px)
DESCRIPTION
TEMPLE: TEMPLate Extension Through Knowledge Acquisition. Yolanda Gil Jim Blythe Information Sciences Institute University of Southern California http://www.isi.edu/expect {gil, blythe}@isi.edu. Acquiring Planning Knowledge. - PowerPoint PPT Presentation
Citation preview
1USC INFORMATION SCIENCES INSTITUTE EXPECT
TEMPLE:TEMPLate Extension Through
Knowledge Acquisition
Yolanda Gil
Jim Blythe
Information Sciences Institute
University of Southern California
http://www.isi.edu/expect
{gil, blythe}@isi.edu
2USC INFORMATION SCIENCES INSTITUTE EXPECT
Acquiring Planning Knowledge Problem: SOF users need to add knowledge to these
planning tools ROEs, commander’s guidance Plan evaluation/critiquing criteria Highlight the information that is important to them Add/extend templates
Approach: provide knowledge acquisition tools to adapt and extend pre-existing planning knowledge Exploit ontologies and background knowledge so users don’t
have to start from scratch KA Scripts guide the user through multiple steps Users manipulate English paraphrases of internal
representations Benefits:
Users can extend the tool’s baseline knowledge for the operation
3USC INFORMATION SCIENCES INSTITUTE EXPECT
Prototype for adding plan critiques: Expect’s PSM Tool
Questions formulatedbased on background
knowledge
User adds detailed knowledge through English paraphrases
4USC INFORMATION SCIENCES INSTITUTE EXPECT
The next 100 days
Allow users to specify and customise “sentinels” that check for new information and alert planners when needed. Our tools generate Java.
Extend ontologies and background knowledge to handle SOF domain.
Integrate with one of the jumpstart applications, probably the travel planning tool, using InterAcT.
5USC INFORMATION SCIENCES INSTITUTE EXPECT
Backup slides
Description of approach, tools and experiment from HPKB project.
6USC INFORMATION SCIENCES INSTITUTE EXPECT
Key Technologies
Guiding users through knowledge acquisition scripts [Tallis and Gil 99] that capture typical dialogues that users follow to enter new knowledge step by step
Exploiting domain-independent background knowledge about plan evaluation and critiquing [Blythe & Gil 99] that use background knowledge about plan evaluation and critiquing to guide the dialog
An English-based editor [Blythe & Ramachandran 99] that lets the user add or modify internal knowledge by manipulating its English paraphrase, without having to see or understand the internal formal representation
7USC INFORMATION SCIENCES INSTITUTE EXPECT
Architecture of TEMPLETemple
(Server)ConstraintAcquisition UI(Client)
Constraintwizard
Constraintviewer
AcquisitionScripts
Background knowledge
Ontologies¤ constraint types¤ actions and plans¤ proactive
suggestions
Method base
Compiler
Executableconstraints
Templatelibrary
Active Templates Toolkit
SOF
Englisheditor
NaturalLanguageGenerator
Domainknowledge
DomainconstraintsDomainmethods
Domainmodelsandtemplates
8USC INFORMATION SCIENCES INSTITUTE EXPECT
Evaluation and Critiquing KnowledgePlan ontology(PLANET)
Ontology ofcritiques
Submethods for checking plan resources
Submethods for checkingplan structure
Reused knowledge(ontologies
and methods)
Domain-specificcritiques
Domain-specific submethods
Domain-specific plan critiquing and evaluation system
Domain-specific knowledge
Ontology ofresources
9USC INFORMATION SCIENCES INSTITUTE EXPECT
An Ontology of Plan Evaluation Criteria
Captures general knowledge of how to evaluate plans with respect to standard norms of plan development
Ill-formeddescription
Statementcritique
Linkcritique
Structurecritique
Complete statement
Correct statement
Clear statementako
ako
ako
Correct linkako Does the purpose of supporting effort X
support the main effort?isa
Does <unit> have sufficient combatpower to accomplish its mission?
isa
Complete planako
...
...isa Does the COA include a statement
of the reserve forces?
10USC INFORMATION SCIENCES INSTITUTE EXPECT
KA Scripts
Helps the user add new critiques by using a background theory of plan evaluation and critiquing.
KA Scripts guide the dialog with the user about the new critique (wizard-type interaction).
The tool creates some of the needed methods for the critiques, helps the user to create new ones (by suggesting initial templates), and ensures consistency with existing knowledge.
11USC INFORMATION SCIENCES INSTITUTE EXPECT
English-Based Editor
Generates automatically English paraphrases of
problem-solving fragments, and presents alternative
text to replace parts of
the paraphrase based
on the ontologies and
background knowledge
NL description of method
Alternatives forselected text fragment
12USC INFORMATION SCIENCES INSTITUTE EXPECT
First experiment: An ablation test on the PSM-Based KA Scripts
Hypothesis: the PSM-Based KA Scripts significantly reduce the expertise and the effort required to add a new critique to the knowledge base.
KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check)
Knowledge (and tool) ablation experiment: Two tasks done using PSM-Based KA Scripts, two tasks done without
Subjects: four Army officers, previously trained on EXPECT’s language for a day
13USC INFORMATION SCIENCES INSTITUTE EXPECT
Sample Tasks Given to Subjects
Simple critique: Add a critique to check if the COA has a security statement.
Complex critique: Add a critique to check if each
task in the COA has sufficient force ratio. To compute force ratio, divide the sum of
combat powers of the Blue units assigned to the task by the sum of combat powers of the Red units acted on by the task.
(Two other comparable tasks were also used)
14USC INFORMATION SCIENCES INSTITUTE EXPECT
Quantitative results: what users could do
Users could complete more tasks using the PSM-based KA scripts
0
5
10
15
20
25
30
35
easiertask 1
easiertask 2
morecomplex
task 1
morecomplex
task 2
All
with PSMToolablated version
LEGEND: indicates total tasks
15USC INFORMATION SCIENCES INSTITUTE EXPECT
Quantitative results: speed improvements
Time reduction using the PSM-based KA Scripts
02468
101214161820
Completenesscritique
(simpler)
Resourcecheck critique
(more complex)
ablatedversionwith scripts
Time inminutes
16USC INFORMATION SCIENCES INSTITUTE EXPECT
Axiom acquisition rates:Experiment with PSM-Based KA Scripts
Adding small amounts of new knowledge
Adding larger amountsof new knowledge
with PSM-BasedKA Scripts
2.12 ax/min
1.26 ax/min
with ablated version
1.1 ax/min
N/A (users were notable to do tasks)
17USC INFORMATION SCIENCES INSTITUTE EXPECT
Summary
Using the PSM-Based KA Scripts significantly reduced the time taken to add a critique
Using the PSM-Based KA Scripts, all four subjects successfully added simple critiques to the EXPECT critiquer; three of them successfully added more complex critiques.
Without the PSM-Based KA Scripts, three out of four subjects successfully added simple critiques and two added more complex critiques.
Comments on the tool usability were positive in all cases.
18USC INFORMATION SCIENCES INSTITUTE EXPECT
Second experiment with PSM-Based KA Scripts and English-Based Editor
Hypothesis: the combination of the PSM-Based KA Scripts and English-based editor allows a user with very little training to add new critiques.
Single subject usability test: A subject was briefed in COA critiquer and the KA interface (but not about EXPECT) for 20min and asked to add two critiques using the tool
KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check), same used in the previous experiment
Subject: an Army officer with no EXPECT training
19USC INFORMATION SCIENCES INSTITUTE EXPECT
Results
The subject was able to add two new critiques of both low and medium complexity.
The time taken was comparable to that for the other four subjects that had previous training in Expect:
02468
101214161820
Completenesscritique
(simpler)
Resourcecheck critique
(more complex)
new user
Average ofother usersTime in
minutes
20USC INFORMATION SCIENCES INSTITUTE EXPECT
EXPECT: A User-Centered Framework for Developing KBSs
Method instantiator
Method instantiator
Knowledge Base
Domain ontologies
and factual knowledge
Problemsolving methods
Domaindependent
KBS
KBS compiler
KBS compiler
Knowledge-BasedSystem
InterdependencyModel (IM)
EXPECT Ontologies and Method libraries
Plans(PLANET)
Evaluations and Critiques
Evaluation PSMs
Resources(OZONE)
COAontologiesCYC/Sensus
Upper
Instrumentation
KA Strategies
KA toolsEMeD
KA Scripts
PSMTool
21USC INFORMATION SCIENCES INSTITUTE EXPECT
EXPECT: A User Centered Approach for Knowledge-Based Planning ToolsKnowledge acquisition technology that can guide users to
specify planning knowledge and develop planning tools Expressive representations
– Loom/Powerloom KR&R– EXPECT’s language to represent problem solving knowledge
Powerful reasoners– Loom/Powerloom pattern classifier & reasoners– abstract problem solving through partial evaluation
ex: how to move <a set of units> from a <location> to another <location>
Explicit models of planning knowledge and plan reasoners: – PLANET ontology of plans, OZONE resource ontology– plan evaluation and planning methods
Expectation-based knowledge acquisition tools– Derive interdependencies between individual knowledge fragments– KA Scripts to guide users in completing complex modifications