CPSC 5307

Search andDiscrete Optimization

Good results with problems that are too bigfor people or computers to solve completely

http://mathworld.wolfram.com/TravelingSalesmanProblem.html

Course outline

textbook – Michalewicz and Fogel(reasonable price, valuable book)

lectures, notes and pptppt presentations evaluation

assignments 5@10% 50% project 10% + 15% + 25% 50%

(documents plus presentations)

Difficult computing problems

hard to representwhat information, what data structurese.g., image organization

no known algorithmse.g., face recognition

no known efficient algorithmse.g., travelling salesperson

This course: discrete variable problems

Good results with problems too big to solve completely

cannot formally optimize results cannot be proven best usually results are not best

how to evaluate results? statistical probability

“within 4%, 19 times out of 20” satisficing

surpassing a threshhold that is “good enough”

Examples practical examples

scheduling (transportation, timetables,…) puzzles

crosswords, Sudoku, n Queens classic examples

SAT: propositional satisfiability problem(independent parameters)

CSP: constraint satisfaction problem(dependent parameters)

TSP: travelling salesman problem(permutations)

SAT: propositional satisfiability problem

P1 P2 P1^P2

F F F

F T T

T F F

T T T

n propositions, P1, P2, P3, …, Pn

What combination of truth values makes a sentence true?

Table has 2n rows.

n=50, 250 = 1,125,899,906,842,624

n=2; 22 = 4 rows

CSP: constraint satisfaction problem

example – map colouringn countries – 4 possible colours- constraints: adjacent

countries different colours- 4n combinations

n=13; 413 = 67,108,864 combinations; 25 constraints

TSP: traveling salesman(sic) problem

n cities: what is shortest path visiting all cities, C1, C2, C3, …, Cn once?

(n-1)! routes from home cityon complete graph

n = 16;(n-1)! = 1,307,674,368,000

C1

n = 5; (n-1)! = 24

Simple Example – one variable

maximize a function, f(n) over a range nmin ≤ n ≤ nmax

f (n) is linear f(n) is monotonic

f (n) is quadratic with f’’(n) < 0 f(n) is convex upward

f (n) is differentiable function f(n) has multiple (local) maxima

Simple Example – one variable

maximize a function, f(n) over a range nmin ≤ n ≤ nmax

f (n) is linear f(n) is monotonic

f (n) is quadratic with f’’(n) < 0 f(n) is convex upward

f (n) is differentiable function f(n) has multiple (local) maxima

Simple Example – one variable

maximize a function, f(n) over a range nmin ≤ n ≤ nmax

f (n) is linear f(n) is monotonic

f (n) is quadratic with f’’(n) < 0 f(n) is convex upward

f (n) is differentiable function f(n) has multiple (local) maxima

Problem description1. fitness function (optimization function, evaluation) –

e.g., m = n3 mod 1012. constraints (conditions) – e.g., n is oddfind global optimum of fitness function withoutviolating constraints ORgetting stuck at local optimum small space:

complete search large space: ?????

Large problems

more possible values more parameters, n = {n1, n2, n3, …} more constraints more complex fitness functions

- takes significant time to calculate m = f(n)

too big for exhaustive search

Searchingwithout searching everywhere

How to search intelligently/efficiently using information in the problem:-hill climbing-simulated annealing-constraint satisfaction-A*-genetic algorithms- …

Focusing search

assumption – some pattern to the distribution of the fitness function (otherwise: random distribution of fitness)

finding the height of land in a forest- can only see ‘local’ structure- easy to find a hilltop but are there other higher hills?

Fitness function distribution

convex – easy – start anywhere, make local decisions

Fitness function distribution many local maxima (conve in local region)

make local decisions but don’t get trapped

Performance – O(f(n))

What is the problem?

Quality software

Correctness Performance == Efficiency

Space – how much memory must be allocated?Time – how much time elapses during

execution?

Measuring performance

Testing Analysis of algorithms

How much space / time is required for an algorithm?

How does one algorithm compare to another?

How do requirements change depending on input?

Need a scale for describing performance

Measuring performance

“Big Oh” notation O(f(n))

Efficiency of performanceexpressed asan approximate function ofthe number of data elements processed

Three simple methodspublic void s0(int n){ int val = n;}

public void s1(int n){ int[] val = new int[n];}public void s2(int n){ int[][] val = new int[n][n];}

n = 6 Memory space

space

We will mainly considerTIME efficiency

Three simple methodspublic void m0(int n){ System.out.println(n);}

public void m1(int n){for(int i = 0; i<n; i++) System.out.println(n);}public void m2(int n){for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}

n = 6# of

executions

1

6

36

time

Three simple methods

public void m0(int n){ System.out.println(n);}

public void m0(int n){ System.out.println(n);}

public void m1(int n){ for(int i = 0; i<n; i++) System.out.println(n);}

public void m1(int n){ for(int i = 0; i<n; i++) System.out.println(n);}

n

time

n

time

n

time

public void m2(int n){ for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}

public void m2(int n){ for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}

O(1)

O(n)

O(n2)

Approximate measurepublic void m2(int n){ for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}public void m2a(int n){ for(int i = 0; i<n; i++) for(int j = i; j<n; j++) System.out.println(n);}

Better performance?

NO

Better performance?

NO

Approximate measure

1 2 6 10 100

# exe

c1 4 36 100 10000

# exe

c1 3 21 55 5050

public void m2(int n){ for(int i = 0; i<n; i++)

for(int j = 0; j<n; j++) System.out.println(n);}

public void m2a(int n){ for(int i = 0; i<n; i++)

for(int j = i; j<n; j++) System.out.println(n);}

n

n2

O(n2)

n2+n 2

O(n2)

Approximate measure

// linear search (unsorted array)public int search(double n, double[] arr){ int i = 0; while (i < arr.length && arr[i] != n) i++; return i;} From 1 to n iterations:

Failed search: n Average for successful search: n/2 Performance O(n)

BUT Different measure// binary search (sorted array)public int search(double n, double[] arr){int i, min = 0, max = arr.length-1; while (min <= max) { i = (min + max ) / 2; if (arr[i] == n) return i; if (arr[i] < n) min = i+1; else max = i-1; } return arr.length;}

From 1 to log n iterations: Failed search: log n Average for successful search: log n - 1 Performance O(log n)

O(log n) performance

n O(n)

n/2 O(n)

log2(n)O(log n)

O(log n) performance

examples: binary search in sorted arrays binary search tree operations

retrieve, update, insert, delete B-trees heaps

insert, remove

Sorting algorithms

two nested repeat operations both O(n) O(n2)

bubble, insertion, selection,… one O(n) and one O(log n) O(n log n)

mergesort, quicksort, heapsort,…

Shellsort: ??, O(n (log n)2), O(n1.25)

Analyzing an algorithm

rule of thumb: depth of nested structure

(or recursion) number of factors

in performance measure

some algorithms are difficult to analyze (e.g. hash table processing)

Comparing ‘polymomial’ performance

nfactors

O(1)0

O(log n)

1O(n)

1O(n log n)

2O(n2)

2O(n3)

3

2 1 1 2 2 4 8

8 1 3 8 24 64 512

32 1 5 64 320 1024 32768

128 1 7 128 896 16384 2097152

1024 1 10 1024 8192 1048576 1073741824

Combinatorial problems

Example: list the possible orderings of S:{1,2,3}: 123, 132, 213, 231, 312, 321

|S| 2 3 4 5 norderings 2 6 24 120 n!

Combinatorial problems

Example: generatetruth table for P^Q:

P Q P^Q

T T T

T F F

F T F

F F F

number of propositions 2 3 4 5 nrows in table 4 8 16 32 2n

Combinatorial problems

only known solutionshave n levels of nestingfor n data values

performance is exponential: O(en) includes n!, 2n, 10n,…

this is our kind of problem!

Comparing polymomial (P) andexponential (NP*) performance

nfactors

O(log n)

1O(n)

1O(n log n)

2O(n3)

3O(en), e.g. 2n

n

2 1 2 2 8 4

8 3 8 24 512 256

32 5 64 320 32768 4294967296

128 7 128 896 2097152 3.4028 x 1038

1024 10 1024 8192 1073741824 1.7977 x 10308

*Non-deterministic Polynomial

NP – Non-deterministic polynomial

if a solution is known, it can be verified in polynomial O(nk) timee.g., SATa solution (T,T,F,T,F,…,T,F,T,F,F,F)can be tested by substituting in the fitness function.

P and NP problems

P = NP?(Are there polynomial algorithms

for combinatorial problems?)YES there are but we are too stupid to find

themNO we need to find other ways to solve

these problems

(in 50+ years of trying, no answer to the question)

NP-hardNP-hard

P and NP problems

NPNP

PP

NP- c

om

ple

teco

mple

te

Representing problems

Abstractionfrom the messy real worldto the ordered simplicity of the model

Abstraction

Representing the problem for processing What information is relevant? How should it be operationalized?

Data Relations Processes

What information is irrelevant?

The puzzles

My real world problem

I have a list of 100 books to order online. Some are available in hardcover (H) or pocket (P) versions and some are available used(U)? How do I minimize my cost for the books?

Variables set of n controllable parameters

V = {x1, x2, x3, …, xn }

each variable xi has a set of possible values, domain Di

e.g., V = { x1, x2, ..., x100 }, 100 books to order

x1 ∈ D1 = {H,P,U}

x2 ∈ D2 = {H,P,U}

...x100 ∈ D100 = {H,P,U}

Problem Space

set of all possible combinations of variable values

dimension of spacen: number of variables

size of space: |D1|x|D2|x…x|Dn|

e.g., book ordersize = 3 x 3 x ... x 3 = 3100

sample point {H, H, U, ..., S}

Fitness

the objective function problem outcome as a function of

the variables: f(x1, x2, x3, …, xn) goal:

optimize (maximize or minimize) fe.g., total cost of purchase f(x1, x2, x3, …, x100)

minimize f

Constraints

rules C(V) that eliminate some points in the problem space from consideration

e.g., some books not available in all versions

Abstraction - “Operationalizing” representation of fitness and constraints for

evaluation and search

e.g. xi = vi = (type, cost, weight)i

fitness f=cost of books plus cost of shippingAll new books (H or S) are shipped in one order at $5 per

kilo of total weight but the shipping cost is waived if the total cost of the new books is over $100. Used books are shipped in a separate order with shipping cost of $3 per book.

How to handle constraints?

Abstraction - “Operationalizing”

e.g. xi = vi =(type, cost, weight)i

fitness f=cost of books plus cost of shippingAll new books (H or S) are shipped in one order at $5 per kilo of total

weight but the shipping rate is waived if the total cost of the new books is over $100. Used books are shipped in a separate order with shipping cost of $3 per book.

How to handle constraints?

cost = ∑ vi.cost // book cost

+ 3 x ∑ (vi.type == U) // ship used

+ [(∑ (vi.type<>U).vi.cost) < 100] // ship new

[5 x ∑ (vi.type<>U).vi.weight]

Exhaustive search

bestFitness = MaxVal (assume minimize)bestVfor all x1 ∈ D1, x2 ∈ D2, x3 ∈ D3,…xn ∈ Dn

if (x1, x2, x3,…xn satisfy constraints C(V))

fitness = f(x1, x2, x3,…xn)if (fitness < bestFitness)

bestFitness = fitnessbestV = {x1, x2, x3,…xn}

return bestFitness, bestV

Exhaustive search example

bestFitness = MaxValbestVfor all x1 ∈ D1, x2 ∈ D2, x3 ∈ D3,…x100 ∈ D100

if (x1, x2, x3,…x100 versions all exist)

cost = f(x1, x2, x3,…x100)if (cost < bestCost)

bestCost = costbestV = {x1, x2, x3,…x100}

return bestCost, bestV

Summary

Parameters, dimension, solution space Objective - fitness or evaluation function Constraints - impossible points in

solution space Finding point in space with optimal fitness

No algorithm to calculate point Space too large to search--> optimization methods with tradeoffs

Puzzles are Us

SEND+MOREMONEY

Each letter represents a different digitNo leading zero’sSum is correctFind assignment of digits to letters