Upload
oswin-poole
View
212
Download
0
Embed Size (px)
Citation preview
RapidKnowledge Formation
Information Extraction and Transport, Inc.with
Veridian PSR,GMU / IB3,
and Paul Cohenunder the direction of
Murray Burke, DARPA
RKF Evaluation Issues
August 2, 2000
RapidKnowledge FormationTKCP Spec
• TKCP spec V1.1 issued
• Y1 content = ECB Chapter 7 (most)
• SQs issued– ECB– Komachi– IET
• Chapters 5 & 6 (portions)• Chapter 7 (reserved)
RapidKnowledge FormationQuestion Ground Rules
• IET’s V1.1 spec is “liberal…”– …w.r.t. KR&R issues– …w.r.t. source “difficulty” ratings
• Teams to identify KR&R “red flags” / issues– Textbook content– SQ-introduced issues & SQ “version space”
• Accepted SQs• Rejected SQs
– “Consensus” ontology of question KR&R issues• Spec includes initial issues assessment for SQs
– Initial feedback milestones: Nov 3, Dec 1
RapidKnowledge FormationPump Priming
• No viable alternatives
• Knowledge for its own sake– Prerequisite, target, fundamental primitives,
common sense– Concepts, relations, axioms
• Knowledge for tools’ sake– Lexical, grammatical…
• Ground rules– “Right” division of labor over KEs, SMEs– Anticipated RKF solutions not ready yet
RapidKnowledge Formation
IET’s SME LabIET’s SME Lab
Process ModelingProcess ModelingProcess ModelingProcess Modeling
E2E Evaluation Strategy
Test Taker(SME)
CPSample Test
CheckpointKB
CheckpointKB
Jun 2000Evaluation
Begins
Aug 2000Evaluation
Ends
Integration Teams
KBPrior
Tools
KB
KBDelta
Integrator KEs
Gatekeeper KE
Sep 2000SpecsReady
RapidKnowledge FormationExperimental Conditions
• User type: KE vs. SME
• KR tool: analogy, sketch, dialog…
• TQ: descriptive, model-based, what-if…
• Pump priming: primitives, prerequisite…