Trust in Multi-Agent Systemsa78khan/docs/CHIL-Oct26_2011.pdf · Atif Khan Oct 26th, 2011 CHIL Meeting. 26/10/11 Atif Khan - CHIL meeting (UoW) 2 Outline Background on Multi-Agent

bySarvapali D. Ramchurn*, Dong Huynh*, and

Nicholas R. Jennings**School of Electronics and Computer Science,

University of Southampton

Trust in Multi-Agent SystemsThe Knowledge Engineering Review,

Vol. 19:1, 1–25 (2004)

Presented by:Atif KhanOct 26th, 2011CHIL Meeting

26/10/11 Atif Khan - CHIL meeting (UoW) 2

Outline

Background● on Multi-Agent Systems (MAS)

My interest● privacy & security of health information

Paper presentation

Trust via Semantic Proof in MAS


Background

Intelligent agent (IA)● autonomous entity

– capable of acting on its own

● interacts with its environment – via observations & actions– goal oriented & utility focused– capable of learning (using knowledge) to achieve goals


Background

Intelligent agent (IA)● rationale

– will act within reason to achieve the goal

● maintains state– personal view of the environment– actions taken over time

● software entities (usually)– at least for our discussion


Background

Multi-Agent System (MAS)● environment of interacting intelligent agents

Characteristics of IA in MAS*● autonomy

– agents are fully/partially autonomous● local views

– agents have local views of the MAS– global view is often not possible

● decentralization– no God agent

*Michael Wooldridge, An Introduction to MultiAgent Systems, John Wiley & Sons Ltd, 2002, paperback, 366 pages


Background

Multi-Agent System (MAS)● MAS are self organizing

– IAs working towards an equilibrium

● decision making– perfect vs. partial information

● communication– protocols: auctions, voting, market, contract-nets– agent communication language (ACL)


Background

Multi-Agent System (MAS)● types of interactions

– cooperative interactions to maximize overall utility,agents share the same utility function

– non-cooperative interactions to maximize self utilityzero-sum games – only a single agent wins (benefits)


Background

Key (agent) interaction problems in MAS● protocol design for multi-agent encounters

● how do agents decide who to interact with

● how do agents decide when to interact with each other


Background

Trust in Multi-Agent System (MAS)


My Interest

Consent-based access control

Hospital B

primary hospital

electronic health

records

consent policyprotection


My Interest


Hospital BHospital A

primary hospital

electronic health

records


treating


My Interest


Hospital BHospital A

who, when and how can health professionals access a patient's recordsas per his consent?

primary hospital

electronic health

records


treating


My Interest

Medical entities as MAS

Institutional

Physician Nurse Patient

Medical RecordCoordinator


My Interest

Medical Entities as MAS

institution

physician nurse patient

medical record coordinator

shift coordinatorsecurity

coordinator

institution

physician nurse patient

medical record coordinator

shift coordinatorsecurity

coordinator


My Interest

Main characteristics● intelligent agents

– are extensions (not replacements) of real world entities

● each institute is an independent MAS– agents are modeled in a hierarchy– agents are allowed to join and leave– utility function → patient utility– one entity-to-one agent (in a MAS)– one entity-to-many agents (one in each MAS)


My Interest

Main characteristics● institutional agents

– form a higher-level MAS– facilitate agents communication across environments

● cooperative interactions– all agents in a single MAS work in a cooperative mode– utility based interaction across environments

can be cooperative or selfish


My Interest

Information Exchange Protocol● allows

– release of patient information on confirmation of consent policy

● multi-party protocol– execution by intelligent agents across

different environments (health MASs)

● trust– we will come back to this


Dr. Sarvapali D. RamchurnIntelligence, Agents, Multimedia Group (IAM)School of Electronics and Computer ScienceUniversity of Southampton (UK)http://users.ecs.soton.ac.uk/sdr/

Dr. Trung Dong HuynhWeb and Internet ScienceSchool of Electronics and Computer ScienceUniversity of Southampton (UK)http://www.ecs.soton.ac.uk/people/tdh

Dr. Nick JenningsAgents, Interaction and Complexity GroupIntelligence, Agents, Multimedia GroupSchool of Electronics and Computer ScienceUniversity of Southampton (UK)http://users.ecs.soton.ac.uk/nrj/

Trust in Multi-Agent SystemsThe Knowledge Engineering Review, Vol. 19:1, 1–25 (2004)


Contribution

Main Contribution● examines trust in multi-agent systems

● provides a literature survey ● evaluates the proposed models

(form the literature)

Key (Agent) Interaction Problem● how to engineer protocols for multi-agent encounters?● how do agents decide who to interact with?● how do agents decide when to interact with each other?


Define Trust First

Definition (DasGupta-98*) ● trust is a belief an agent has that the other party will do

what it says it will

*Dasgupta, P. 1998 Trust as a commodity. In Gambetta, D. (ed.), Trust: Making and Breaking Cooperative Relations. Blackwell, pp. 49–72.


Define Trust First

Definition (DasGupta-98*) ● trust is a belief an agent has that the other party will do

what it says it will

Other Definitions†

*Dasgupta, P. 1998 Trust as a commodity. In Gambetta, D. (ed.), Trust: Making and Breaking Cooperative Relations. Blackwell, pp. 49–72.

†Donovan Artz, Yolanda Gil. 2007 A survey of trust in computer science and the Semantic Web. Web Semantics: Science, Services and Agents on the World Wide Web, Vol. 5, No. 2. (June 2007), pp. 58-71.


Trust

Individual Trust System Trust

learning & evolution

reputation models

socio-cognitive models

truth eliciting protocols

reputation mechanisms

security mechanisms

Classification of Trust Components


Two Approaches to Trust

Individual Level Trust● an agent trusting other agents● uses trust models to reason about other agent's

– reciprocative nature– reliability– honesty

● trust model– utilized to calculate the degree of trust an agent

can place in interactions with an other agent– is built over multiple interactions


Two Approaches to Trust

System Level Trust● protocols & mechanisms that force an agent to be

trustworthy while interacting with other agents● usually enforced via

– protocol design (e.g. auctions)example: English auction

– cryptographic primitives

Observation● the two approaches are complementary to each other


Individual Level Trust

Motivation● an agent (situated in an open environment)

– trying to choose the most reliable interaction partner– deciding the (interaction) strategy to adopt– requires a trust model


Individual Level Trust

Aspects of (individual level) Trust● learning & evolving

– strategies, trust metrics● reputation models● socio-cognitive models


Learning & Evolving Trust● “trust as an emergent property of direct interactions

between self-interested agents”

Model assumptions● agents will interact with each other many times● agents have an incentive to defect

– to get higher payoffs– however, reduces future interactions with losing agents

● agents know the payoff of each encounter– i.e. an agent can choose the best possible move based on

inferred best possible move of the opponent (game theory – von Neuman & Morgenstern 1944)


Strategies – ( Learning & Evolving Trust)

Tit-For-Tat (TFT) strategy● cooperates on the first move and then

imitates the opponent's move for all remaining interactions

● Axelrod's “Prisoner's dilemma” tournaments (84)– showed (within very controlled settings) that

tit-for-tat strategy was most successful*as compared to other selfish or nicer strategies

– won by eliciting cooperationnot by doing better

*most successful = higher average point over the tournaments



Tit-For-Tat (TFT) strategy● observations

– generally tit-for-tat induces trust between agents– would punish untrustworthy behavior– would forgive if trustworthy behavior is shown again– leads to highest payoff when interacting

parties adhere to the protocol

– maximum gain is not possible in the presence of other selfish strategies

● potential to loose on the first encounter



Strategy selection (Wu & Sun 2001)● agents to adapt (their strategies) to the environment

– minimize losses and foster cooperation

● trust emerges as agents evolve a trusting relationship – by evaluating the benefit of each possible strategy over

multiple interactions

● they showed that:– tit-for-tat behavior + evolution of strategies

→ allows nice agents to beat nasty agents in the long run– ignored the cost of cooperation



Strategy selection (Sen 1996)● cost of cooperation:

– utility lost on the first move – initially ignored (by Wu & Sun)

● demonstrates:– “reciprocity can emerge when the

agents learn to predict that they will receive future benefits if they cooperate”



Guidelines for evolutionary stable strategiesin multi-strategy environments (Sen & Dutta 2002)● observations

– collaborative liars perform well when the number of interactions is small & the number of cooperating agents is high

– in all other scenario reciprocative strategy performs better

– length & number of interactions mattercorroborated by Mui et al. (2002)probabilistic model which defines a threshold for the number of encounters required for trust



Trust establishment with future knowledge (Mukherjee et al. 2001)● trust based on knowing opponent's chosen move in

advance – generally based on agent preferences

● observations– if an agent does not reveal or partially reveal their actions

before the opponent moves, then no trust can be established, as the opponent will always defect

– in a bilateral information exchange scenarioagents (are forced) trust each other through mutually learning to choose the best strategy (max gain)

● cost of interaction & returns form future actions



Trust in cooperative environments● not all multi-agent interactions are strictly competitive

● agents are allowed to be self interested but still need to achieve a maximum payoff for the group– e.g. common fund for building a road

each agent contributes money to a common pottotal amount collected establishes if the road can be built



Trust in cooperative environments (Birk 2000, 2001)● N-prisoner's dilemma as base model● agents contribute to a common fund

required for a social gain● an agent may be tempted to under contribute


Trust Metrics – ( Learning & Evolving Trust)

Measuring trust● computational trust modeling

– rate the performance of an agent– historical tracking of this performance

Witkowski et al. (2001): trust model● trust is calculated based on the

performance in the past interactions● trust via measurable quantities (resource consumption)



Witkowski et al. (2001): trust model● trading model where bandwidth is traded● inter-agent trust → quality and quantity of traded

bandwidth● trust calculation based on type of agent

– consumer agent: ● “update their trust value according to the difference

between their bids and the received goods (bandwidth)”

– supplier agent:● “supplier agents update their trust in the consumers

according to the extent to which the quality (size) of the goods (bandwidth) supplied has been exploited”



Bi-stable value models(Mui 02, Sen & Sajja 02, Schillo 00)● performance of an agent is

either good or bad

General Observation● works well in simulated settings

but lacks the richness required for real world settings asrealistic interactions involve richer outcomes– e.g. quality of goods traded, efficiency of task handling,

duration of task



REGRET Sabater & Sierra (2002)● “gives richer semantics to ratings (or impressions) by

defining their particular characteristics”

● impression measurement of agent actions on per characteristic of an interaction

example:an agent can express a satisfaction −0.5 for the delivery date of some goods and +1 for the price of the same goods



REGRET Sabater & Sierra (2002)● overall trust computed based on

all impressions measured using fuzzy reasoning techniques

● impressions are taken based on – individual interactions– agent's system wide interactions

susceptible to strategic liars


Trust



reputation models




security mechanisms


Reputation Modeling

Definition (Sabater & Sierra 02)● “view of someone about something”

– where view → aggregation of opinions of the members of the community

Aspects of reputation modeling● methods to gather ratings using existing social

relationships, defining trustworthiness of an agent● reliable reasoning methods to reason about the

“aggregation of opinions”


Gathering Rating (Reputation Modeling)

Social Network● “to organise the retrieval and aggregation of ratings

from other agents, most reputation models borrow the concept of a social network” (Burt, 1982; Buskens, 1998)

● Assumptions– agents are related to each other

via roles or communication links– agents as witnesses of interactions,

can transmit information about each other (Panzarasa et al., 2001)



Gathering rating from social network● referrals (Yu & Singh 2000)

– referrals are pointers to other sources of information– agents explore starting with their neighbors and

gradually build the social network of trustworthy agents

● add agent characteristics to the mix (Schillo et al. 2000)– each node holds two values

a) degree of honesty of an agentb) degree of altruism (selflessness)

– these values used to calculate trustworthiness of a witness



Deduction of Higher Level Concepts● “neighbors (group)”● example

– “Yu and Singh’s model takes into account ratings from those agents that are

close (by virtue of the number of links separating them with a potential interaction partner) to choose witnesses for a particular agent.

Underlying this is the assumption that closer witnesses will return more reliable ratings”


Aggregating Rating (Reputation Modeling)

eBay● simple “+1 / -1” aggregation rating● no penalty for having no ratings

– can be unreliablewhen buyers do not return ratings



Yu & Singh 02 (referrals)● deal with the absence of information

information gathering (referrals based) + information aggregation (Dempster Shafer theory of evidence)– “it allows one to combine evidence from different sources and

arrive at a degree of belief (represented by a belief function) that takes into account all the available evidence”



Yu & Singh 02 (referrals)● Dempster’s rule allows the combination of beliefs

obtained from various sources(saying an agent is trustworthy, untrustworthy, or unknown to be trustworthy or not)to be combined so as to support the evidence that a particular agent is trustworthy or not

● if personal rating based on direct interactions present then disregard referrals

● does not deal with lying



Problem of lying witnesses● an agent may lie about its ratings of another agent

Schillo et al. (2000)● decompose the rating into (a)trust & (b)altruism● assign probability to a witness lying

– is learned over multiple interactionsrecursive aggregation over the network taking into consideration the probability that the witnesses queried may lie

● trust value is more reliable than Yu & Singh's model



Problem of lying witnesses● an agent may lie about its ratings of another agent

Sen et al. (2000)● via learning (rather than subjective probabilities)● sharing of trust values can benefit reciprocative agents in

the long run● selfish & lying agents may win for short runs● overtime, colluding agents can not exploit reciprocative

agents● requires a threshold number of witnesses to work



Sabater & Sierra 02 ● social dimension of reputation

reputation value → weighted sum of– individual subjective impressions– group impressions– group impression of agent's group– individual impression of the agent's group

● older ratings are given less importance● ratings are obtained in a cooperative manner



Sabater & Sierra 02 ● ontological dimension

example– a travel agent being good might imply

low price for one agent, but may imply good quality seats reserved for another

● aggregation method is susceptible to:– lying & noise


Trust



reputation models




security mechanisms


Socio-Cognitive Trust Models

Cognitive view of trust – Castelfranchi & Falcone (98, 00, 01)● include environment and opponent characteristics

agent x needs to delegate a task to agent y, agent x evaluates trust based on beliefs● competence: y is indeed capable of carrying out the task● willingness: y has decided and is willing to do the task● persistence: y is stable enough to complete the task● motivation: y has some motives to help x


Socio-Cognitive Trust Models

Observations● different characteristics have different impact

on the overall trust value– example:

competence belief is a prerequisitemotivational belief can vary based on future payment

● impact of global knowledge of beliefs– example

what happens when agent y knows that agent x trusts it?would it also increases the trustworthiness of x for y?


Semantic proofs & multi-agent trust


Semantic Proofs

:John a :Patient; :hasPolicy :optin.:HIV_MR a :MedicalRecord; :belongsTo :John.:DrSmith a :Physician; :isTreating :John.

{?P :haspolicy :optin.?MR :belongsTo :?P.?DOC :isTreating ?P} {?⇒ DOC :hasAccess ?MR}.

_:WHO :hasAccess :HIV_MR.

{{:John :hasPolicy :optin} e:evidence <knowledgebase#_2>. {:HIV_MR :belongsTo :John} e:evidence <knowledgebase#_4>. {:DrSmith :isTreating :John} e:evidence <knowledgebase#_6>} =>

{{:DrSmith :hasAccess :HIV_MR} e:evidence <rules#_1>}.}.

KnowledgeStore

Rule

Query

Reasoner

Proof

Result


Semantic Proofs

Characteristics of a Semantic Proof● first order logic proof

● verifiable by any rational party– verified by traversing the knowledge graph

and applying the inference rules

● provides confidence in the result– the proof path in the knowledge graph tells

you how the proof was reached

● provides auditing capabilities


Example Scenario

Request for Information● Dr request for P's medical record from H1● H2 (institutional agent) propagates the request to H1● H1 (institutional agent) receives and processes the

request

H2(CGH)

Dr

PH1(TGH)


Example Scenario

Proof Generation● H1 (TGH) identifies protection set PS

– PS {patient consent C, H1 privacy & security policies H1Policy}

● H1 requests H2 (CGH) for provable validation of PS– C & H1Policy

● H2 generates the proof and returns to H1

H1P

EMR

SC1 H2 SC2

P'

C & H1Policy Proof


Example Scenario

Required Proofs

Consent policyOpt-out with emergency override

Required Proof:●confirm that patient is indeed in an emergency situation

TGH security & privacy policy - must be an employee - must be treating the patient - must be on shift - must be a physician

Required Proof:●DR is an employee of the hospital●DR is treating the patient●DR is on shift●DR is a physician


Example Scenario

Proof Validation● H1 computes the proof

– locally – using a trusted third party proof checker

● Information is exchanged – upon successful validation of proof

(of consent & other policies)


Example Scenario

Trust Establishment● on per request basis● via successful semantic proof validation


Trust via Semantic Proof

Characteristics● assumes no previous trust relationship● trust is calculated and verified for each action● trust for each action can be weighted differently

as we know the semantics of the game● can be used for agent-to-agent and

agent-to-system level trust establishment● can be verified by any third party● neutral to system level protocols and ACLs● facilitates intra-MAS agent trust establishement


Trust via Semantic Proof

Characteristics● depends on data provenance

– possible to construct a valid from bad dataso that the proof validation will be successful

– can use cryptographic primitives to ensuredata integrity & provenance

● can withstand and deal with Sybil agents● can be used with existing ACLs

– exchanging proof's is just exchanging data

Documents

Trust in Multi-Agent Systemsa78khan/docs/CHIL-Oct26_2011.pdf · Atif Khan Oct 26th, 2011 CHIL Meeting. 26/10/11 Atif Khan - CHIL meeting (UoW) 2 Outline Background on Multi-Agent