7.8 What is an algorithm?Intractability...7 Literal.A Boolean variable or its negation. Clause.A disjunction of 3 distinct literals. A propositional formula # that is the conjunction

Introduction to Computer Science • Robert Sedgewick and Kevin Wayne • Copyright © 2006 • http://www.cs.Princeton.EDU/IntroCS

7.8 Intractability

2

Overview

Q. What is an algorithm?

A. Definition formalized using Turing machines.

Q. Which problems can be solved on a computer?

A. Computability.

Q. Which algorithms will be useful in practice?

A. Analysis of algorithms.

Q. Which problems can be solved in practice?

A. Intractability.

3

Q. Which algorithms are useful in practice?

A working definition. [von Neumann 1953, Godel 1956, Cobham 1960, Edmonds 1962]

! Measure running time as a function of input size N.

! Efficient = polynomial time for all inputs.

! Inefficient = "exponential time" for some inputs.

Ex. Dynamic programming algorithm for edit distance takes N2 steps.

Ex. Brute force algorithm for TSP takes N! steps.

Theory. Definition is broad and robust; huge gulf between polynomial

and exponential algorithms.

Practice. Exponents and constants of polynomials that arise are small !

scales to huge problems.

Properties of Algorithms

4

Exponential Growth

Exponential growth dwarfs technological change.

! Suppose each electron in the universe had power of today's

supercomputers…

! And each works for the life of the universe in an effort to solve

one TSP problem via brute force.

! Will not help solve 1,000 city TSP problem via brute force.

Quantity

Supercomputer instructions per second

Age of universe in seconds †

Number

1013

1017

Electrons in universe † 1079 † Estimated

1000! >> 101000 >> 1079 " 1017 " 1013

5

P

Def. P is the set of all yes-no problems solvable in poly-time on a

deterministic Turing machine.

Remark. Algorithm typically also solves the related search problem.

34, 5134, 39Euclid (300 BCE)Are x and y relatively prime?RELPRIME

5351Agarwal-Kayal-

Saxena (2002)

Does x have a factor other

than 1 and itself?COMPOSITE

acgggt

ttttta

niether

neither

Dynamic

Programming

Is the edit distance between

strings x and y less than 5?

EDIT-

DISTANCE

Is there a vector x that

satisfies Ax = b?

Description

Gauss-Edmonds

elimination

Algorithm

LSOLVE

Problem NoYes

!

0 1 1

2 4 "2

0 3 15

#

$

% % %

&

'

( ( (

,

4

2

36

#

$

% % %

&

'

( ( (

!

1 0 0

1 1 1

0 1 1

"

#

$ $ $

%

&

' ' '

,

1

1

1

"

#

$ $ $

%

&

' ' '

6

Extended Church-Turing Thesis

Extended Church-Turing thesis.

! P = yes-no problems solvable in poly-time in this universe.

! If computable by a piece of hardware in time T(N) for input of size

N, then computable by TM in time (T(N))k for some constant k.

Evidence supporting thesis.

! True for all physical computers.

! k = 2 for random access machines. [your laptop]

Implication. To make future computers more efficient, only need to

focus on improving implementation of existing designs.

Law of physics? A new constraint on what is possible?

Possible counterexample? Quantum computers. like 2nd law of thermodynamics

7

Literal. A Boolean variable or its negation.

Clause. A disjunction of 3 distinct literals.

Conjunctive normal form. A propositional

formula # that is the conjunction of clauses.

3-SAT. Given a CNF formula # consisting of k clauses over n literals,

does it have a satisfying truth assignment?

3-Satisfiability

!

Cj = x1" x

2" x

3

!

xi or x

i

!

" = C1#C

2# C

3# C

4 # C

5

!

x1" x

2" x

3( ) # x1" x

2" x

3( ) # x1" x

2" x

3( ) # x1" x

2" x

4( ) # x2" x

3 " x

4( )

!

x1 = true, x2 = true, x3 = false, x4 = true

a satisfiable formula with 5 clauses and 4 variables

8

Q. How to solve an instance of 3-SAT.

A1. Try all 2n truth assignments.

A2. Do something substantially more clever???

Conjecture. No poly-time algorithm for 3-SAT.

Intractability of 3-SAT

"intractable"

9

Reductions

Q. Which problems won't we be able to solve in practice?

A. No easy answers, but theory helps.

Def. Problem X polynomial reduces to problem Y if arbitrary instances

of problem X can be solved using:

! Polynomial number of standard computational steps, plus

! One call to subroutine for Y as last step.

Consequence. If no poly-time algorithm for X, then no poly-time

algorithm for Y.3-SAT

your research problem

10

Planar 3-Color

Given a planar map, can it be colored using 3 colors so that no adjacent

regions have the same color?

YES instance.

11

Planar 3-Color

Given a planar map, can it be colored using 3 colors so that no adjacent

regions have the same color?

NO instance.

12

3-Colorability

3-COLOR. Given a graph (need not be planar), is there a way to color

the vertices red, green, and blue so that no adjacent vertices have the

same color?

Applications. Register allocation, Potts model in physics, …

yesinstance

13

3-Colorability

3-COLOR. Given a graph, is there a way to color the vertices

red, green, and blue so that no adjacent vertices have the same color?

14

3-Colorability

Claim. 3-SAT polynomial reduces to 3-COLOR.

Pf. Given 3-SAT instance #, we construct an instance of 3-COLOR that

is 3-colorable iff # is satisfiable.

Construction.

i. Create one vertex for each literal.

ii. Create 3 new vertices T, F, and B; connect them in a triangle,

and connect each literal to B.

iii. Connect each literal to its negation.

iv. For each clause, attach a gadget of 6 vertices and 13 edges.

to be described next

15

3-Colorability

Claim. Graph is 3-colorable iff # is satisfiable.

Pf. ! Suppose graph is 3-colorable.

! Consider assignment that sets all T-colored literals to true.

! (ii) ensures each literal is same color as T or F.

! (iii) ensures a literal and its negation are opposites.

true false

baseB

T F

!

x1

!

x1

!

x2

!

x2

!

xn

!

xn

!

x3

!

x3

16

3-Colorability






! (iv) ensures at least one literal in each clause is same color as T.

T F

!

Ci

= x1V x

2V x

3

6-node gadget

true false

B

!

x1

!

x2

!

x3

17

3-Colorability






! (iv) ensures at least one literal in each clause is same color as T.

T F

B

!

x1

!

x2

!

x3

not 3-colorable if all are red

true false

contradiction!

Ci

= x1V x

2V x

3

6-node gadget

18

3-Colorability


Pf. $ Suppose 3-SAT formula # is satisfiable.

! Color all true literals same color as T.

! Color vertex below green node F, and vertex below that B.

! Color remaining middle row vertices B.

! Color remaining bottom vertices T or F as forced. !

T F

B

!

x1

!

x2

!

x3

a literal set to true in 3-SAT assignment

true false

!

Ci

= x1V x

2V x

3

6-node gadget

19

More Poly-Time Reductions

3-SAT

3DM VERTEX COVER

HAM-CYCLECLIQUE

INDEPENDENT SET

3-COLOR

PLANAR-3-COLOREXACT COVER

HAM-PATHSUBSET-SUM

PARTITION INTEGER PROGRAMMING

KNAPSACK

Dick KarpTuring award (1985)

3-SAT

reduces to 3-COLOR

TSP

BIN-PACKINGConjecture: no poly-time algorithm for 3-SAT.

(and hence none of these problems)20

Still More Hard Computational Problems

Aerospace engineering. Optimal mesh partitioning for finite elements.

Biology. Protein folding.

Chemical engineering. Heat exchanger network synthesis.

Chemistry. Chemical synthesizability.

Civil engineering. Equilibrium of urban traffic flow.

Economics. Computation of arbitrage in financial markets with friction.

Electrical engineering. VLSI layout.

Environmental engineering. Optimal placement of contaminant sensors.

Financial engineering. Find minimum risk portfolio of given return.

Game theory. Find Nash equilibrium that maximizes social welfare.

Genomics. Phylogeny reconstruction.

Mathematics. Given integer a1, …, an, compute

Mechanical engineering. Structure of turbulence in sheared flows.

Medicine. Reconstructing 3-D shape from biplane angiocardiogram.

Operations research. Traveling salesperson problem.

Physics. Partition function of 3-D Ising model in statistical mechanics.

Politics. Shapley-Shubik voting power.

Pop culture. Minesweeper consistency.

Statistics. Optimal experimental design.

6,000+ scientific papers per year.

21

Implications of Intractability

Proving a problem intractable guide scientific inquiry.

! 1926: Ising introduces simple model for phase transitions.

! 1944: Onsager find closed form solution to 2D case in tour de force.

! 19xx: Feynman and other top minds seek 3D solution.

! 2000: Istrail reduces 3-SAT to 3D version of problem.

a holy grail of statistical physics

search for closed formula appears doomed

22

Cook's Theorem

3-SAT

3DM VERTEX COVER

HAM-CYCLECLIQUE

INDEPENDENT SET

3-COLOR


HAM-PATHSUBSET-SUM


KNAPSACK

3-COLOR

reduces to 3-SAT

TSP

BIN-PACKINGAll of these problems (any many more)polynomial reduce to 3-SAT.

Stephen CookTuring award (1982)

23

Cook + Karp

3-SAT

3DM VERTEX COVER

HAM-CYCLECLIQUE

INDEPENDENT SET

3-COLOR


HAM-PATHSUBSET-SUM


KNAPSACK

3-COLOR

reduces to 3-SAT

TSP

BIN-PACKINGAll of these problems are different manifestations

of one "really hard" problem.

3-SAT

reduces to 3-COLOR

24

Complexity Classes

P. Set of yes-no problems solvable in poly-time on a deterministic TM.

EXP. Same as P, but in exponential-time.

NP. Same as P, but on non-deterministic Turing machine.

Cook-Levin Theorem (1960s). ALL NP problems reduce to 3-SAT.

NP-complete. All NP problems to which 3-SAT reduces.

Implications.

! If efficient algorithm for 3-SAT, then P = NP.

! If efficient algorithm for any NP-complete problem, then P = NP.

! If no efficient algorithm for some NP problem, then none for 3-SAT.

if we can solve 3-SAT,we can solve any of them

if we can solve any ofthem, we can solve 3-SAT

25

Certificates

Alternate (and Equivalent) Definition. NP is set of all yes-no problems

for which you can check in poly-time that it is a yes instance (given a

certificate).

3-COLOR. Given a graph, is there a way to color the vertices

red, green, and blue so that no adjacent vertices have the same color?

instance certificate

26

Certificates

Alt. Def. NP is set of all yes-no problems for which you can check in

poly-time that it is a yes instance (given a certificate).

FACTOR. Given two integers x and y, does x have a factor

between 2 and y?

x = 2773

y = 50

instance

certificate

47 ! 59

x = 147573952589676412927

y = 147573952589676412927

193707721 ! 761838257287

instance

certificate

267 - 1

27

The Main Question

Does P = NP?

! Is there a poly-time algorithm for SAT?

! Does nondeterminism help you solve problems faster?

! Clay $1 million prize.

If yes: Efficient algorithms for 3-COLOR, TSP, FACTOR, . . .

If no: No efficient algorithms possible for 3-COLOR, TSP, . . .

Consensus opinion on P = NP? Probably no.

would break modern cryptographyand collapse economy

EXP NP

P

If P % NP If P = NP

NP-complete

EXP

P = NP

Jack Edmonds, 1962

28

Implications of NP-Completeness

Classify problems according to their computational requirements.

! NP-complete: SAT, all Karp problems, thousands more.

! P: RELPRIME, COMPOSITE, LSOLVE.

! Unclassified: FACTOR is in NP, but unknown if NP-complete or in P.

Computational universality.

! All known algorithms for NP-complete problems are exponential.

! If any NP-complete problem proved exponential, so are rest.

! If any NP-complete problem proved polynomial, so are rest.

Proving a problem is NP-complete can guide scientific inquiry.

! 1926: Ising introduces simple model for phase transitions.

! 1944: Onsager solves 2D case in tour de force.

! 19xx: Feynman and other top minds seek 3D solution.

! 2000: Istrail proves 3D problem NP-complete.

31

Coping With Intractability

Relax one of desired features.

! Solve the problem in poly-time.

! Solve the problem to optimality.

! Solve arbitrary instances of the problem.

Complexity theory deals with worst case behavior.

! Instance(s) you want to solve may be "easy."

! Concorde algorithm solved 13,509 US city TSP problem.

32






Develop a heuristic, and hope it produces a good solution.

! No guarantees on quality of solution.

! Ex: TSP assignment heuristics.

! Ex: Metropolis algorithm, simulating annealing, genetic algorithms.

Design an approximation algorithm.

! Guarantees to find a nearly-optimal solution.

! Ex: Euclidean TSP tour guaranteed to be within 1% of optimal.

! Active area of research, but not always possible!

Sanjeev Arora (1997)

33






Exploit intractability.

! Cryptography. [stay tuned]

Keep trying to prove P = NP.

can do any 2 of 3

34

Summary

Many fundamental problems are intractable.

! TSP, 3-SAT, 3-COLOR.

! 3D Ising model.

Theory says: we probably won't be able to design efficient algorithms

for them.

! You will encounter such problems in your scientific lives.

! If you know about intractability, you can identify them and

avoid wasting time and energy.

Documents

7.8 What is an algorithm?Intractability...7 Literal.A Boolean variable or its negation. Clause.A disjunction of 3 distinct literals. A propositional formula # that is the conjunction