KNOWLEDGE ACQUISITION II

Protocol Analysis

• Knowledge provider works through some pre-defined task, KE observes.

• Varieties:– Think-aloud– Critical response– Periodic report– Report by commentary

Think-aloud protocol

• Knowledge provider performs task, “thinking aloud” as s/he does it

keeps elicitation on track, but:time consuming

knowledge provider may forget

knowledge provider unlikely to verbalise at appropriate grain level

Critical response

Knowledge provider gives description when sub-task completed

likely to generate a summarised set of necessary sub tasks

rule derivation may therefore be less time consuming

but:

summary information only

holes in the knowledge therefore possible

Periodic report

Knowledge provider reports, at given time intervals, what s/he is trying to achieve, and the steps leading to it

useful if nature of task precludes vocalising and working simultaneously

keeps provider’s concentration up

but time consuming

Report by commentary

Ask knowledge provider to watch video, and comment

General danger with protocol analysis - observer effect GOW

(“Hawthorne effect”)

Possible guidelines for PA sessions

• (Based on Schreiber et al 2000):– Present problems and data in a realistic way– Transcribe the protocols as soon as possible– Avoid long self-report sessions– Be present, even during think-aloud sessions

Multi-dimensional scaling

• Collect objects in the domain• Have expert manipulate these, and hence sort

into related groups according to different criteria(hence “multi-dimensional”) GOW

•• Hope thus to achieve understanding of the

domain ontology• Varieties:

– Card sort– Repertory grid

Card-sort

Essential idea:• obtain set of concepts that covers the requisite

domain• transfer each to a card• ask expert to sort each into groups, and identify

what each has in common• iteratively combine the groups to form a

hierarchy• shuffle cards and repeat

“Advantages” of card sort technique

• Quick and relatively easy way of establishing the global structure of a set domain area

• Easy to perform - KE has little involvement while cards being sorted

• Relevance assured• Can help uncover implicit knowledge

(Milton et al 1999)

“Disadvantages” of card sort technique

• Time/effort to prepare the cards• Danger of gaps in the hierarchical model -

assumes KE has got the cards right• (wise therefore to have some blank cards)• Debatable how much used:

– “virtually unused” (Wagner et al, 2003 p 82, but cf. ibid. p 80!)

– “invaluable part of the knowledge engineer’s toolkit” (Rugg and McGeorge, 2005 p 105).

– arguably useful re information system design, especially web sites (eg Robertson, 2001)

Repertory grids

Essential idea:• obtain set of concepts that covers the requisite domain• transfer each to a card• expert draws three at random• expert says which two are the most alike• says why (“identifies the construct” behind the decision)• repeat until reasonable number of constructs• expert then rates each object according to its position on

a bipolar scale for each of the constructs

“Advantages” of repertory grids

• can elicit similarities and subtle distinctions between concepts (Wagner et al 2003)

• computational tools available (Rugg and McGeorge 2005)

• extensively used (ibid.)

• can help uncover implicit knowledge (ibid.)

“Disadvantages” of repertory grids

• may require specialist statistical skill by the KE

• very demanding for provider, if many comparisons

• approach is of no use if data do not form semantic scales (ibid.)

Laddered grid

Aim: gain spatial representation of relationships between domain concepts (“domain ontology”), and information about definitions.

Approach:start at arbitrary point in the hierarchical concept space (the “seed” item)work up, down across the space, gaining information about superordinate and subordinate concepts

Hence 4 types of question likely to be asked:“What is <concept> an example of?” - probing for superordinate concepts“Can you give examples of <concept>?” - probing for subordinate concepts“Are there any other examples of <concept>?” - probing for concepts at same level

as named concept“What distinguishes <concept1> from <concept2>?” - probing for criterial attributes

Grid is drawn as the session proceeds

“Advantages” of laddered grids

• exploration of a domain can be quite rapid, and engineer obtains easily interpreted domain map

• may be able to move directly from ladder to rules and/or semantic net - no transcription

• little domain knowledge required by KE• can create hierarchies of knowledge elements

such as concepts, attributes, processes and requirements

• A “well established knowledge acquisition technique” (Yan et al 2005)

“Disadvantages” of laddered grids

• people differ in their ability to think “spatially”

• “a somewhat contrived technique” (Schreiber et al 2000)

• assumes domain of interest is hierarchically structured (Yan et al 2005)

Other approaches to knowledge acquisition

• Forward scenario simulation• Goal decomposition

– (similar - chapter listing)

• Inquisitive observation• Systematic symptom to fault links• 20 questions• Review • Machine learning

Summary

• Knowledge acquisition is difficult• But crucial for expert system development

– And arguably for “general” AI– Unless machine learning can come to the fore

• Various techniques for knowledge acquisition exist – and arguably are crucial re “knowledge management”– and also for requirements analysis (Rugg and

McGeorge 2005)• The techniques are by no means mutually

exclusive (ibid.)

More exam information

• There are 3 questions

• Question 1 is worth 40 marks (half of the exam) and was given out last week

• Question 2 contains 5 parts, each worth 4 marks

• Question 3 contains 3 parts (worth 6, 6 and 8).

Exam information – topics covered

• Introduction • Expert systems• Neural networks• Multi-Criteria Decision Modelling in Business• Knowledge Representation (I) – production rules• Knowledge Representation (II) – logic and frames• Uncertainty and risk• Decision Modelling: A Survey of Business Models• Knowledge Acquisition and Management I• Knowledge Acquisition and management II• KBDSS: Ethics, Business and Society I• KBDSS: Ethics, Business and Society II

Exam information – topics not in the exam

• Nothing on the final 2 weeks apart from question 1 (the one you already have)

• Nothing on uncertainty

• Nothing on MCDM

Ergo – what to revise

• The philosophy question• Expert systems (2 very interesting

lectures)• Neural networks• Knowledge rep - rules, frames, logic (2

very interesting lectures)• Survey of business models• Knowledge acquisition (2 very interesting

lectures)