Relational Relational Preference for Preference for

ControlControlRonen BrafmanRonen Brafman

Department of Computer Department of Computer Science Ben-Gurion Science Ben-Gurion

UniversityUniversity

Decision Making Contexts: Decision Making Contexts: Two ExtremesTwo Extremes

1.1. Online applicationsOnline applications Lay usersLay users Preference elicitation interface must be based on Preference elicitation interface must be based on

very simple natural language expressionsvery simple natural language expressions Users will spend at most a few minutes, usually Users will spend at most a few minutes, usually

lessless

2.2. Decision analysis contextDecision analysis context Facilitated by a human expertFacilitated by a human expert Complex modeling that requires intelligent Complex modeling that requires intelligent

framingframing Quantitative modelQuantitative model Considerable effortConsiderable effort

A Third Context: System A Third Context: System ControlControl

Goal: Design a system that behaves well in Goal: Design a system that behaves well in a large and diverse number of settings that a large and diverse number of settings that cannot all be foreseen offline. cannot all be foreseen offline.

A generic preference model is used by the A generic preference model is used by the system to select among alternative choicessystem to select among alternative choices

The designer is willing to spend time The designer is willing to spend time constructing the modelconstructing the model

Yet, the language must be intuitive so that:Yet, the language must be intuitive so that: The specification process is convenient and The specification process is convenient and

accessibleaccessible The specification is easy to modifyThe specification is easy to modify

Decision-Theoretic System Decision-Theoretic System DesignDesign

Not a new ideaNot a new idea See Russell and Norvig’s AI: A Modern See Russell and Norvig’s AI: A Modern

ApproachApproach Recent successful autonomous vehiclesRecent successful autonomous vehicles

A purist approach:A purist approach: Maintain a probability distribution, Maintain a probability distribution,

utility functionutility function Select actions that maximize expected Select actions that maximize expected

utilityutility

Decision-Theoretic System Decision-Theoretic System Design: Utilities Get the Design: Utilities Get the

Short EndShort End Work on probabilistic reasoning is far Work on probabilistic reasoning is far

aheadahead Relational and object-oriented Relational and object-oriented

probabilistic modelsprobabilistic models Inference algorithmsInference algorithms Learning methodsLearning methods

Good progress on the utility/preference Good progress on the utility/preference specification side, but not on par with specification side, but not on par with the probabilistic modelsthe probabilistic models

Some Possible ReasonsSome Possible Reasons

Historic lagHistoric lag

Machine learningMachine learning Probably the most important application Probably the most important application

area of AIarea of AI A lot of dataA lot of data Great need for modeling itGreat need for modeling it

What About Preference What About Preference Models?Models?

Modeling preferences is harder:Modeling preferences is harder: It is harder to introspect about preferencesIt is harder to introspect about preferences It is harder to learn preferences – little data if anyIt is harder to learn preferences – little data if any

Can we learn a preference model for a new application?Can we learn a preference model for a new application?

Still - as system become more complicated, Still - as system become more complicated, the need for tools that guide them is more the need for tools that guide them is more pressingpressing

We need to provide designers with We need to provide designers with convenientconvenient tools for describing complex tools for describing complex control preferencescontrol preferences

Rule-Based SystemsRule-Based Systems

Rule-based systems are intuitiveRule-based systems are intuitive Are often used in industry to Are often used in industry to

describe complex control rulesdescribe complex control rules Are very rigid:Are very rigid:

If the body is true, so is the headIf the body is true, so is the head Certain contexts (constraints) can Certain contexts (constraints) can

render them inconsistentrender them inconsistent Context sensitivity must be built in Context sensitivity must be built in

carefullycarefully Limited modularityLimited modularity

Utility FunctionsUtility Functions

Immediate context awarenessImmediate context awareness If some constraint holds, seek best If some constraint holds, seek best

feasible solutionfeasible solution Additive forms (GAI) provide Additive forms (GAI) provide

modularitymodularity Defined over a rigid set of Defined over a rigid set of

propositions propositions

work with a rigid, predefined set of work with a rigid, predefined set of objectsobjects

Relational/OO Preference Relational/OO Preference RulesRules

Simple and intuitive like rule-based Simple and intuitive like rule-based systemssystems

Induce a utility/value function for Induce a utility/value function for any given set of objectsany given set of objects

Simple additive semantics provides Simple additive semantics provides modularitymodularity

A Concrete ApplicationA Concrete Application

Command and Control Monitoring:Command and Control Monitoring: High ranking officials monitoring masses of High ranking officials monitoring masses of

data in a real-time command & control data in a real-time command & control centercenter

Our goal: decide which data to show at Our goal: decide which data to show at each point in timeeach point in time

Video streamsVideo streams Sensor dataSensor data Results of relevant queriesResults of relevant queries Results of data analysis (e.g., simulations, risk Results of data analysis (e.g., simulations, risk

assessment)assessment)

Example : Fire Example : Fire DepartmentDepartment

Cameras on fireman helmet, fixed Cameras on fireman helmet, fixed surveillance camerassurveillance cameras

Heat sensors, smoke detectors, coHeat sensors, smoke detectors, co22--levelslevels

Area maps, building plans, driving Area maps, building plans, driving distance, number of residentsdistance, number of residents

Simulation of structure strength, time Simulation of structure strength, time to contain a fire as function of wind and to contain a fire as function of wind and other weather conditions, etc.other weather conditions, etc.

Requirements: Object Requirements: Object Oriented and Generic Oriented and Generic

SpecificationSpecification Fire departments in different cities Fire departments in different cities

have different personal, equipmenthave different personal, equipment Objects within a single department Objects within a single department

change:change: New firemenNew firemen New equipmentNew equipment Fire events – naturally modeled as an objectFire events – naturally modeled as an object

One preference specification should One preference specification should cover allcover all

Modeling a Fire Modeling a Fire Department - IDepartment - I

Object classes: fireman, fire-engine, Object classes: fireman, fire-engine, fire, camera, heat-sensor, cofire, camera, heat-sensor, co22-sensor-sensor

Fireman attributesFireman attributes Name, Location, Rank, Role, …Name, Location, Rank, Role, … Camera, heat-sensor, …Camera, heat-sensor, …

Camera attributes:Camera attributes: OnOn DisplayDisplay URLURL

Modeling a Fire Modeling a Fire Department - IIDepartment - II

Rules:Rules:Fireman(x) Fireman(x) ΛΛ fire(y) fire(y) ΛΛ x.location = x.location =

y.location y.location

x.camera.display x.camera.display [“on” 4, “off” 0][“on” 4, “off” 0]

Fireman(x) Fireman(x) ΛΛ x.co x.co22-level=high -level=high

x.camera.display x.camera.display [“on” 8, “off” 0][“on” 8, “off” 0]

More FormallyMore Formally

Model includes:Model includes: Set of object classesSet of object classes

Each class specifies object attributes and Each class specifies object attributes and their type (and possibly class attributes)their type (and possibly class attributes)

Set of preference rules of the form:Set of preference rules of the form: rule-body rule-body rule-head [ rule-head [vv11, w, w11; v; v22 w w22;…v;…vmm w wmm]]

Preference RulesPreference Rules

rule-body rule-body rule-head [ rule-head [vv11, w, w11; v; v22 w w22;…v;…vmm wwmm]]

Rule-body: classRule-body: class11((xx11) ) ΛΛ … … ΛΛ class classkk((xxkk) ) ΛΛ αα1 1 ((xxi1i1)) ΛΛ… … Λ Λ ααs s ((xxisis))

ααl l ((xxll)) : : xxii.path REL .path REL xxjj.path OR .path OR xxii.path REL value.path REL value REL: <, >, =, ≠, etc.REL: <, >, =, ≠, etc. Path: an attribute chain, such as x.mother.professionPath: an attribute chain, such as x.mother.profession

Rule-head: Rule-head: xxii.path.pathvvi i – – a possible value of a possible value of xxii.path.pathwwi i – – a real numbera real number

Controllable and Controllable and Uncontrollable AttributesUncontrollable Attributes

We distinguish between controllable We distinguish between controllable attributes, to which we need to assign a attributes, to which we need to assign a valuevalue Example: camera.displayExample: camera.display

And uncontrollable, or context attributesAnd uncontrollable, or context attributes Example. Fire.locationExample. Fire.location

The body may refer to both controllable and The body may refer to both controllable and uncontrollable attributesuncontrollable attributes

The head contains a single controllable The head contains a single controllable attributeattribute

Informal SemanticsInformal Semantics

Intuitive semantics: the value of value Intuitive semantics: the value of value vvi i forfor

xxii.path is .path is wwi i whenwhen αα11 ΛΛ… … Λ Λ ααl l are satisfied A form of conditional, valued preference

If you want, a relational UCP-net Preferences rules r + set of object

instances o + assignment to uncontrollable (context) attributes

utility function over all possible assignments to the controllable attributes of o.

Obtaining the Value Obtaining the Value FunctionFunction

Step 1: Given a set of objects, and a set of Step 1: Given a set of objects, and a set of rules, generate all ground instances of the rules, generate all ground instances of the rulesrules

Step 2: Filter rule bodies based on the Step 2: Filter rule bodies based on the value of context attributes:value of context attributes: Remove satisfied conjunctsRemove satisfied conjuncts Eliminate rules with false conjunctsEliminate rules with false conjuncts

Step 3: We now have a set of ground rules Step 3: We now have a set of ground rules containing only controllable attributescontaining only controllable attributes

Step 4: The value of assignment Step 4: The value of assignment ππ to the to the uncontrollable attributes is the sum of all uncontrollable attributes is the sum of all weights associated with heads of weights associated with heads of unassigned rules whose body unassigned rules whose body ππ satisfies satisfies

Example Example Consider the rules we saw before:Consider the rules we saw before:

fireman(x) fireman(x) ΛΛ fire(y) fire(y) ΛΛ x.location = y.location x.location = y.location x.camera.display [“on” 4, “off” 0]x.camera.display [“on” 4, “off” 0]

Fireman(x) Fireman(x) ΛΛ x.co x.co22-level=high -level=high x.camera.display [“on” 8, “off” 0]x.camera.display [“on” 8, “off” 0]

Objects: fireman(Alice)Objects: fireman(Alice) Fireman(Alice) Fireman(Alice) ΛΛ Alice.co Alice.co22-level=high -level=high

Alice.camera.display [“on” 8, “off” 0]Alice.camera.display [“on” 8, “off” 0] Objects: fireman(Alice),fire(fire1)Objects: fireman(Alice),fire(fire1)

fireman(Alice) fireman(Alice) ΛΛ fire(fire1) fire(fire1) ΛΛ Alice.location = Alice.location = WTC.location WTC.location

Alice.camera.display [“on” 4, Alice.camera.display [“on” 4, “off” 0]“off” 0]

Fireman(Alice) Fireman(Alice) ΛΛ Alice.co Alice.co22-level=high -level=high Alice.camera.display [“on” 8, “off” 0]Alice.camera.display [“on” 8, “off” 0]

Example - FilteringExample - Filtering Fireman(Alice) Fireman(Alice) ΛΛ Alice.co Alice.co22-level=high -level=high

Alice.camera.display [“on” 8, “off” 0]Alice.camera.display [“on” 8, “off” 0] Alice.coAlice.co22-level=high -level=high True True Alice.camera.display [“on” 8, “off” 0] Alice.camera.display [“on” 8, “off” 0] Alice.coAlice.co22-level=low -level=low ǾǾ

fireman(Alice) fireman(Alice) ΛΛ fire(fire1) fire(fire1) ΛΛ Alice.location = Alice.location = fire1.location fire1.location

Alice.camera.display [“on” 4, “off” 0]Alice.camera.display [“on” 4, “off” 0] Alice.location = wtc; fire1.location = wtcAlice.location = wtc; fire1.location = wtc True True Alice.camera.display [“on” 4, “off” 0] Alice.camera.display [“on” 4, “off” 0] Alice.location ≠ WTC. LocationAlice.location ≠ WTC. Location ǾǾ

Example Example Objects: fireman(Alice), Alice.coObjects: fireman(Alice), Alice.co22-level=high-level=high

True True Alice.camera.display [“on” 8, “off” 0] Alice.camera.display [“on” 8, “off” 0] Value(Alice.camera.display = on) = 8Value(Alice.camera.display = on) = 8 Value(Alice.camera.display = off) = 0Value(Alice.camera.display = off) = 0

Objects: fireman(Alice),fire(fire1), Alice.coObjects: fireman(Alice),fire(fire1), Alice.co22--level=high, Alice.location = fire1.location = level=high, Alice.location = fire1.location = wtcwtc True True Alice.camera.display [“on” 8, “off” 0] Alice.camera.display [“on” 8, “off” 0] True True Alice.camera.display [“on” 4, “off” 0] Alice.camera.display [“on” 4, “off” 0]

Value(Alice.camera.display = on) = 12Value(Alice.camera.display = on) = 12 Value(Alice.camera.display = off) = 0Value(Alice.camera.display = off) = 0

(a bit) More Formally(a bit) More Formally A variable may appear only if it also appears A variable may appear only if it also appears

within a unary “class” variable.within a unary “class” variable. Constrains the number of ground instances to be finite Constrains the number of ground instances to be finite Finite universe also implies finite number of attributesFinite universe also implies finite number of attributes

Filtered ground rules induce value function over Filtered ground rules induce value function over the set of assignments to controllable attributesthe set of assignments to controllable attributes

vvR,O R,O (a)(a) – value of assignment a given rules – value of assignment a given rules rr objects objects oo r’ – r’ – ground instances of rules in ground instances of rules in rr W(r’,a) – W(r’,a) – the weight associated with the rule head of r’ the weight associated with the rule head of r’

and assignment aand assignment a

'

, ),'()(r

OR arwav

Some Modeling IssuesSome Modeling Issues

CompositionalityCompositionality Should we allow multiple rules with identical Should we allow multiple rules with identical

heads?heads? Our answer – yesOur answer – yes Fits naturally with the idea of additive Fits naturally with the idea of additive

representationrepresentation Example:Example:

fireman(x) fireman(x) ΛΛ fire(y) fire(y) ΛΛ x.location = y.location x.location = y.location x.camera.display [“on” 4, “off” 0]x.camera.display [“on” 4, “off” 0]

fireman(x) fireman(x) ΛΛ fireman(y) fireman(y) ΛΛ fire(z) fire(z) ΛΛ x.location = y.location x.location = y.location ΛΛ x.location = z.location x.location = z.location ΛΛ x ≠ y x ≠ y ΛΛ x.camera.display = x.camera.display = “on” “on” y.camera.display [“on” -4, “off” 0] y.camera.display [“on” -4, “off” 0]

More Modeling IssuesMore Modeling Issues Indirect preferencesIndirect preferences

I want to say “low-ranking drivers are I want to say “low-ranking drivers are preferred for fire-engines”preferred for fire-engines”

I could write:I could write: Fire-engine(x) Fire-engine(x) x.driver.rank [“low” 4, high “0”] x.driver.rank [“low” 4, high “0”]

But rank is not a controllable attribute – But rank is not a controllable attribute – driver is!driver is!

Violates our restriction on rule-headsViolates our restriction on rule-heads Maybe: fire-engine(x) Maybe: fire-engine(x) ΛΛ fireman(y) fireman(y) ΛΛ y.rank y.rank

= low = low x.driver=y [T 4, F 0]? x.driver=y [T 4, F 0]? Somehow, more flexibility is desirable.Somehow, more flexibility is desirable.

Computing Optimal Computing Optimal AssignmentAssignment

Branch and boundBranch and bound Local searchLocal search

May be good for real-time applications May be good for real-time applications with many computations with similar with many computations with similar sets of objectssets of objects

Reduction to relational probabilistic Reduction to relational probabilistic modelsmodels

Related WorkRelated Work

Relational probabilistic modelsRelational probabilistic models Close links between probabilities/utilities/valuesClose links between probabilities/utilities/values

All are forms of orderingsAll are forms of orderings Multiplicative decomposition of joint distribution Multiplicative decomposition of joint distribution

similar to additive decomposition of value functionsimilar to additive decomposition of value function Just take the logarithmJust take the logarithm

There are various prm formalismsThere are various prm formalisms Syntax is similarSyntax is similar Semantics is similar in the sense that prm + objects Semantics is similar in the sense that prm + objects

induces a distribution over assignment to Herbrand induces a distribution over assignment to Herbrand BaseBase

Compositionality more of an issueCompositionality more of an issue Inference is also mostly based on grounding firstInference is also mostly based on grounding first

Related WorkRelated Work

Relational Models – translationRelational Models – translation Given: rule-body Given: rule-body rule-head [ rule-head [vv11, w, w11; v; v22

ww22;…v;…vmm w wmm]] Markov Logic: rule-body Markov Logic: rule-body ΛΛ rule- rule-

head(head(vvii) : ) : wwii

Relational Bayesian Logic: Pr(rule-head = vi | rule-body) = exp(wwii))

Renormalize weights to ensure Renormalize weights to ensure exp(wwii) ) in [0,1]in [0,1] Use MAP queries to find optimumUse MAP queries to find optimum

Related WorkRelated Work

Soft Constraint Programs – more Soft Constraint Programs – more generalgeneral Preference rules attempt to provide a Preference rules attempt to provide a

minimalist solution that generalizes GAI minimalist solution that generalizes GAI value functionvalue function

Ongoing WorkOngoing Work C&C system implementation on-goingC&C system implementation on-going

Prototype completion expect in JulyPrototype completion expect in July Will test B&B vs. local searchWill test B&B vs. local search Future plans – try to use Alchemy (engine for Future plans – try to use Alchemy (engine for

Markov Logic)Markov Logic) Integrate with uncertaintyIntegrate with uncertainty

May not be hard given the relation with PRMsMay not be hard given the relation with PRMs May be viewed as some form of relational May be viewed as some form of relational

influence diagram where decisions are not influence diagram where decisions are not orderedordered

Cyclic dependencies!Cyclic dependencies! Try to learn from observationTry to learn from observation

Based on user interaction with the system, try to Based on user interaction with the system, try to learn/modify ruleslearn/modify rules

SummarySummary

Very simple formalismVery simple formalism Attempt to minimally extend rule-based Attempt to minimally extend rule-based

systems to define value functionssystems to define value functions Much similarity to existing Much similarity to existing

approaches for PRM and soft approaches for PRM and soft constrain programmingconstrain programming

Can utilize algorithms from PRMCan utilize algorithms from PRM