CPSC 322, Lecture 35Slide 1 Finish VE for Sequential Decisions & Value of Information and...

CPSC 322, Lecture 35 Slide 1

Finish VE for Sequential Decisions

& Value of Information and Control

Computer Science cpsc322, Lecture 35

(Textbook Chpt 9.4)

April, 3, 2009

CPSC 322, Lecture 35 Slide 2

Lecture Overview

•Sequential Decisions•Optimal Policy

•Variable Elimination

•Value of Information•Value of Control

CPSC 322, Lecture 35 Slide 3

Sequential decisions (Planning)under uncertainty

• What has been the output of planning so far?

• Why this is not adequate?

• What should the output be?

A sequence of actions

The agent does not know how the environment will be when it will have to decide (i.e., to select an action)

For each decision (action selection) , a specification of what action will be the best in each possible configuration of the environment

Complexity of finding the optimal policy: how many policies?

• If a decision D has k binary parents, how many assignments of values to the parents are there?

• If there are b possible actions, how many different decision functions are there?

• If there are d decisions, each with k binary parents and b possible actions, how many policies are there?

• How many assignments to parents?

• How many decision functions? (binary decisions)

• How many policies?

Finding the optimal policy more efficiently: VE

1. Remove all variables that are not ancestors of the utility node

2. Create a factor for each conditional probability table and a factor for the utility.

3. Sum out random variables that are not parents of a decision node.

4. Eliminate (aka sum out) the decision variables

5. Sum out the remaining random variables.

6. Multiply the remaining factors: this is the expected utility of the optimal policy.

Eliminate the decision Variables: details• Select a variable D that corresponds to the latest

decision to be made• this variable will appear in only one factor with its parents

• Eliminate D by maximizing. This returns:

• The optimal decision function for D, arg maxD Value

• A new factor to use in VE, maxD Value

• Repeat till there are no more decision nodes.

Report CheckSmoke Value

true truetrue falsefalse truefalse false

-5.0 -5.6 -23.7 -17.5

Example: Eliminate CheckSmoke

Report CheckSmoke

true false

Report

Value

truefalse

New factor

Decision Function

VE elimination reduces complexity of finding the optimal policy

• We have seen that, if a decision D has k binary parents, there are b possible actions, If there are d decisions,

• Then there are: (b 2k)d policies

• Doing variable elimination lets us find the optimal

policy after considering only policies (we eliminate one decision at a time)

• VE is much more efficient than searching through policy space.

• However, this complexity is still doubly-exponential we'll only be able to handle relatively small problems.

CPSC 322, Lecture 35 Slide 8

Lecture Overview

•Sequential Decisions•Optimal Policy

•Variable Elimination

•Value of Information•Value of Control

CPSC 322, Lecture 33 Slide 9

Value of Information

• What would help the agent make a better Umbrella decision?

• The value of information of a random variable X for decision D is:

the utility of the network with an arc from X to D minus the utility of the network without the arc.• Intuitively:• The value of information is always

• It is positive only if the agent changes

CPSC 322, Lecture 33 Slide 10

Value of Information (cont.)• The value of information provides a bound on how much you should be prepared to pay for a sensor. How much is a perfect weather forecast worth?

• Original maximum expected utility:• Maximum expected utility when we know Weather:• Better forecast is worth at most:

77

91

CPSC 322, Lecture 33 Slide 11

Value of Information• The value of information provides a bound on how much you should be prepared to pay for a sensor. How much is a perfect fire sensor worth?

• Original maximum expected utility:• Maximum expected utility when we know Fire:• Perfect fire sensor is worth:

-22.6

-2

CPSC 322, Lecture 35 Slide 12

Lecture Overview

•Sequential Decisions•Optimal Policy

•Variable Elimination

•Value of Information•Value of Control

CPSC 322, Lecture 33 Slide 13

Value of Control

• The value of control of a variable X is :the utility of the network when you make X a

decision variable minus the utility of the network when X is a random variable.

• What would help the agent to make an even better Umbrella decision? To maximize its utility. Weather Umbrella Value

Rain trueRain falsenoRain truenoRain false

700 20100

CPSC 322, Lecture 35 Slide 14

Value of Control

• What if we could control the weather?

• Original maximum expected utility:• Maximum expected utility when we control the weather:• Value of control of the weather:

77

100

CPSC 322, Lecture 33 Slide 15

Value of Control• What if we control Tampering?

• Original maximum expected utility:• Maximum expected utility when we control the Tampering:• Value of control of Tampering:• Let’s take a look at the policy

• Conclusion: do not tamper with fire alarms!

-22.6

-20.7

CPSC 322, Lecture 4 Slide 16

Learning Goals for today’s (and Wed) classes

You can:• Represent sequential decision problems as

decision networks. And explain the non forgetting property

• Verify whether a possible world satisfies a policy and define the expected value of a policy

• Compute the number of policies for a decision problem

• Compute the optimal policy by Variable Elimination

• Compute value of information and control

Next class

• Markov Decision Processes (MDPs) (textbook 9.5)

Assignment 4 due on Wed Apr 8

• Need to talk to student 521320