Temporal Knowledge Acquisition From Multiple Experts

Temporal Knowledge Acquisition From Multiple Experts

Temporal Knowledge Acquisition From Multiple Experts

by Helen Kaikova, Vagan Terziyan

The problem of knowledge acquisition

The problem of knowledge acquisition

T T1 2

Event 1 Event 2

DOMAINTemporal

relation

Most supported opinion

Experts

Experts'

ranking

1. How to derive the most supported knowledge from the multiple experts about the unknown temporal relation between the two events?

2. How to make quality evaluation of the most supported opinion?

3. How to make semantic analysis of the most supported opinion to make necessary corrections?

4. How to make, evaluate, use and refine ranking of all the experts to improve the results?

Basic concepts: Basic relationsBasic concepts: Basic relations

M

Group 1 Group 2

M1 X-< Y

- M4 X-< Y

+

M2 X-> Y

- M5 X-> Y

+

M3 X-= Y

- M6 X-= Y

+

Group 3 Group 4

M7 X+< Y

- M10 X+< Y

+

M8 X+> Y

- M11 X+> Y

+

M9 X+= Y

- M12 X+= Y

+

T1

T2

Relation between endpoints

M - the set of m basic relations between temporal endpoints of two intervals [ X-, X+] and [ Y-, Y+]

Basic concepts: Experts and Ranks Basic concepts: Experts and Ranks

with ranks

S - the set of n knowledge sources or experts

S1 S2 S3 S4

R1 R2 R3 R4

BeforeT

T1

2T T1 2

AfterT

T1

2T T1 2

R 1

R 2

Met-byT

T1

2T T1 2 R 4

MeetsT1

2T T1 2 R 3

T

DuringT

T1

2T T1 2 R 5

T2

T1 IncludesT T1 2 R 6

OverlapsT

T1

2T T1 2

OverlappedT

T1

2 T T1 2

R 7

R 8

Started-byT

T

1

2

T T1 2 R 10

StartsT1

2T T1 2 R 9

T

FinishesT

T1

2T T1 2 R 11

T2

T1 FinishedT T1 2 R 12

by

by

T2

1 T T1 2 R 13EqualsT

Basic concepts: RelationsBasic concepts: Relations

R - the set of r basic relations for temporal intervals

(Allen’s basic temporal relations)

T1

T2

Relation:

T1 Overlaps T2

Relation between endpoints

Basic conceptsBasic concepts

R M - t h e m a t r i x r × m d e f i n e s r e l a t i o n s h i p b e t w e e n t h e

r e l a t i o n s ’ s e t s R a n d M b y t h e f o l l o w i n g w a y :

R R M M R M R Mk q k q k q, ( ( ) ), ;

R M X- < Y

-X

- > Y-

X- = Y

- … X+ = Y

+

R 1 1 0 0 … 0

R 2 0 1 0 … 0

… … … … … …

R 1 3 0 0 1 … 1

Basic concepts Basic concepts

T - the set of t temporal intervals which correspond to the domain events

The Domain

Event 1 Event 2

Event 3

T1

T3

T2

TemporalIntervals

??

?

Basic concepts: Semantic predicateBasic concepts: Semantic predicate

P - the semantic predicate which defines piece of knowledge about temporal relationship by the following relation between the sets T, R and S:

P T R T S

if the knowledge source S

uses R to describe relation

between T and T

otherwise

a k b i

i

k

a b( , , , )

,

;

, .

1

0

Basic Concepts: Basic Concepts:

SMa,b - the matrix n×m with expert knowledge about

two fixed temporal intervals Ta and Tb; it defines relationship between the set of knowledge sources S and set of basic relations M by the following way:

T T T S S M M R

R P T R T S R M SM

a b i q k

a k b i k q i qa b

, , , ,

(( ( , , , )&( )) ).,,

Refinement strategyRefinement strategy

•All experts have the same initial rank, equal to one.

•After each vote the rank of each expert should be recalculated.

•An expert improves his rank after some vote if his opinion has less conflicts with the most supported one than the average number of conflicts among all the experts. Otherwise he loses some part of his rank. The responsibility of each expert for the results of his voting should grow from vote to vote.

•An expert’s rank should not be changed after some vote if expert does not participate it or his opinion has as many conflicts with the most supported one as the average number of conflicts among others.

•An expert’s rank should always be more than zero and less than n (number of experts).

•The value of punishment (or prize) for presence (or absence) of each conflict should be maximal for expert whose rank is equal to n/2.

•The value of punishment (or prize) for presence (or absence) of each conflict should be aspire to zero for expert whose rank is close to zero or to n.

Ranking votes refinement techniqueRanking votes refinement technique

rvi - is the rank of i-th expert before v-th vote

rvi - is the rank of i-th expert before

v-th vote

0 r niv

r i i ni1 1 1 , ( , . . . , )

VOTE a,b - vector which contains results of the current experts’

voting concerning relation between intervals Ta and Tb derived

from the matrix SM a,b as follows:

VOTE abs

a b t q m

qa b

qa b

qa b, , ,( ),

, , , ,

1 1

where qa b

iv

ii SM

nr

i qa b

,

,( ),

,

1

qa b

iv

ii SM

nr

i qa b

,

,( ),

,

0

The most supported opinionThe most supported opinion

MSUP a,b - is the vector which contains most supported opinion

concerning relation between intervals Ta and Tb derived as follows:

( ) ,

, , , ,

, , , qa b

qa b

qa bMSUP

a b t q m

0

1 1

Correction rulesCorrection rules

1. If <all the components from the same group of the most supported opinion are equal to zero>,

then <that one which has the least vote should be changed to one>.

a b t h h

MSUP q Group

s Group VOTE

VOTE MSUP

qa b

h

h sa b

Groupa b

sa b

h

, , , ,

, (( , )&

&( (

min( )))) .

,

,

, ,

1

1 4


2. If <all the components from the same group of the most supported opinion are equal to zero and more then one of them has minimal vote in the group>,

then <that one which corresponds to the relation of equivalence between temporal points should be changed to one>.

a b t h h Group

q q q MSUP q Group

VOTE VOTE VOTE OR

OR VOTE VOTE

VOTE MSUP

h

qa b

h

qa b

qa b

qa b

qa b

qa b

qa b

qa b

, , , , , ((

, , )&( , )&

&((( ) )

(( )

))) .

,

, , ,

, ,

, ,

1 1 4

1 2 3

1 3 2

2 3

1 3


3. If <there are more than one components from the same group of the most supported opinion which are equal to one>,

then <that one which has the least vote should be changed to zero>.

a b t h h MSUP

MSUP q s Group VOTE

VOTE MSUP

sa b

qa b

h sa b

qa b

sa b

, , , , , (( &

& , , )&(

)) .

,

, ,

, ,

1 1 4


4. If <there are more than one equal to one components from the same group of the most supported opinion including the last one and they have the same vote>,

then <those ones which are not correspond to the relation of equivalence between temporal points should be changed to zero>.

a b t h h Group q q q

s Group q MSUP MSUP

VOTE VOTE MSUP

h

h qa b

sa b

qa b

sa b

sa b

, , , , , (( , , )&

&( )&( & &

&( ))) .

, ,

, , ,

1 1 4 1 2 3

33

3

Refinement of Expert ranksRefinement of Expert ranks

r r riv

iv

iv 1

rcon

coniv

iv

iv

viv

convi - is the number of

conflicts between opinion of i-th expert and the most supported opinion calculated through all set M during the v-th vote:

con SM MSUP

a b t i n v

iv

i qa b

qa b

q

m

( ),

, , , , , , ,

,, ,

1 1 1

iv i

vivr n r

n

( )

1

i n

1

where vjv

j

n

ncon

1,

Most supported opinion “quality” evaluationMost supported opinion “quality” evaluation

Q Votes accepted as most supported opinion

All votes

Q

VOTE

m rva b

qa b

q

m

iv

i

n,

,

ProgramProgram

Experts’ votes on 3 problems

Resulting Most supported opinion

Consensus’dynamics

Few tasks canbe maintained

Qualities

Starting

The reverse example (first vote)The reverse example (first vote)

Old ranks New ranks

Most supported on this stage

Endpoints’ relations

Ranks’ Changing During The ExampleRanks’ Changing During The Example

0

0.51

1.5

2

2.53

3.5

4

1 11 21

Experts' votings

Exp

ert

s' ra

nks

0

0.2

0.4

0.6

0.8

1

S 1

S 2

S 3

S 4

Quality

MSUP: 12,1,8

Qu

ali

ty

Ranks’ Changing During The Reverse Example

Ranks’ Changing During The Reverse Example

0

0.51

1.5

2

2.53

3.5

4

1 11 21

Experts' votings

Exp

ert

s' ra

nks

0

0.2

0.4

0.6

0.8

1

S 1

S 2

S 3

S 4

Quality

MSUP: 7,3,12

Qu

ali

ty

ConclusionConclusion

The method can also be used to derive the most supported knowledge in domains that have a well-defined set of basic relations (such as the set M) and a set of compound relations (such as the set R). In those cases, experts are allowed to give incomplete or incorrect knowledge about a relation from the set R because it can be handled by deriving the most supported opinion of multiple experts through the set M.

Ranking technique in this research supports experts whose opinion is close to the most supported one. It even may happen that only one expert remains after several votes whose rank becomes more than all other ranks together. In many applications, the most supported knowledge is not the best one. Moreover the talent individuals can easily lose their rank if they are thinking not like others. That is why method should be developed to pick up and classify experts whose opinions are the most different from the most supported opinion, and then try to take them into account.

Documents

Temporal Knowledge Acquisition From Multiple Experts