An Introduction to Computational Complexity

Edith ElkindIAM, ECS

Knapsack Problem

• n items, each item has– a weight wi

– a value vi

• Knapsack capacity W• Target value V• Question: can we pack the knapsack

to reach the target value? – is there a vector (x1, ..., xn) {0, 1} n such that

w1 x1 + ... + wn xn ≤ W v1 x1 + ... + vn xn ≥ V

Knapsack Problem: Special Cases

• Subset Sum: – for each item weight = value– capacity of knapsack = target value– Question:

is there a vector (x1, ..., xn) {0, 1} n such that w1 x1 + ... + wn xn = W

• Partition:can we split a given list into 2 equal-weight parts? – like Subset Sum, but additionally w1 + ... + wn = 2W

Subset Sum is NP-complete

• Reduction from 1-in-3-SAT– n variables, m clauses– can you set the variables so that

in each clause, exactly one variable is satisfied?x1 V ¬x2 V x3, x2 V x3 V x4:

x1=T, x2=T, x3=F, x4=F YESx1 V x2 V x3, x1 V ¬x2 V x3, ¬ x1 V x2 V x3: x1=T => contradiction x1=F => contradiction

NO

Subset Sum is NP-complete

• Proof:– 2n numbers: 2 per variable (one for xi, one for ¬xi)

– each number has 2(n+m) binary digits – digits 2i-1, 2i encode variable i

0 0 1000 0 0 0xi

¬xi

2i-1 2i

0 0 1000 0 0 0

0 1 1010 ... ... ...W

forced to pickexactly one ofxi , ¬xi

2n

.........

...

... ...

Subset Sum is NP-complete

• Proof:– 2n numbers: 2 per variable– each number has 2(n+m) digits – digits 2n+2j-1, 2n+2j encode clause j

0 0 10...... ... ... ...x1

¬x3

2j-1 2j

0 0 10...... ... ... ...

0 1 10...... ... ... ...

0 0 10...... ... ... ...

W

x7

forced to pickexactly one ofx1 , ¬x3, x7

cj = x1 V ¬x3 V x7

2n

Subset Sum is NP-complete

• Proof:– 2n numbers: 2 per variable– each number has 2(n+m) digits – digits 2i-1, 2i encode variable i– digits 2n+2j-1, 2n+2j encode clause j– pick exactly one of the numbers for xi , ¬xi

– can obtain W iff exactly one variable per clause is satisfied

What about Partition and Knapsack?

• Subset Sum is a special case of Knapsack =>Knapsack is NP-hard

• Can reduce Subset Sum to Partition:– instance of Subset Sum: (w1 , ..., wn ; W)

– set X = w1 + ... + wn

– instance of Partition: (3X-W, 2X+W, w1 , ..., wn ) • yes-instance of SS => yes-instance of P:

3X-W + subset of wi s of weight W has weight 3X2X+W + subset of wi s of weight X-W has weight 3X

• yes-instance of P => yes-instance of SSany set of weight 3X contains exactly one of 3X-W, 2X+W

Knapsack: an (Efficient?) Algorithm• Dynamic programming:

V(i, w): maximum value you can achieve by using a subset of the first i items of weight w– V(1, w): 0 if w ≠ w1 and v1 if w = w1

– V(i+1, w): max {V(i, w), vi+1 +V(i, w - wi+1)}

0 0 0050 0 0 00 3 8050 0 0 0

item 1: w1 = 4, v1 = 5

item 2: w2 = 2, v1 = 3

... x y......... ... ... ...

... ... z......... ... ... ...z = max {y, vi+1 + x}

wi+1

i=2

i=1

i+1

i

Knapsack: an (Efficient?) Algorithm

• V(i, w): maximum value you can achieve by using a subset of the first i items of weight w

• V(n, w): maximum value of a subset of weight w• if V(n, w) ≥ V for some w ≤ W,

the answer is “yes”, else “no”• Running time:

– each V(i, w) can be computed in log V time– nW values to compute– final scan: O(W) O(nW log V)

What Is an Efficient Algorithm?

• Knapsack input:– n+1 numbers of size at most W each– n+1 numbers of size at most V each– representation size: O(n(log W + log V))

• numbers are represented in binary

• Dynamic programming algorithm: O(nW log V)– exponential in input size:

W is exponentially bigger than log W

Small Numbers

• Knapsack input:– n+1 numbers of size at most W each– n+1 numbers of size at most V each– representation size: O(n(log W + log V))

• What if weights an values are small?– W = poly(n), V = poly(n)– DP running time: O(nW log V) = poly (n):

polynomial in input size!

Binary vs. unary notation

• Knapsack input:– n+1 numbers of size at most W each– n+1 numbers of size at most V each

• What if inputs are given in unary? – w is represented as 111.....111

– representation size: O(n(W + V))

• DP running time: O(nW log V): polynomial in input size!

w times

Strong vs. Weak NP-hardness

• A problem is weakly NP-hard, if it is hard when inputs are given in binary, but not in unary

• A problem is strongly NP-hard, if it is hard even when inputs are given in unary

• Example: Longest Path– input: Graph G=(V, E), source s, sink t, target k, edge

lengths l1, …, l|E|

– question: is there a loop-free path from s to t of length at least k?

– hard even for small weights (in fact, 0/1 weights)

Implications

• If you have an NP-hard problem, try to figure out where the hardness comes from:– big numbers in the problem?– structure of the problem?– both?

• Useful for understanding which special cases are likely to be easy

Coping With Intractability

• Good algorithm:(1) returns exact answer(2) works on all instances(3) runs in poly-time

• For NP-hard problems, can have at most 2 out of 3!– (1)+(2): exp-time algorithms (can be practical)– (1)+(3): heuristics– (2)+(3): approximation algorithms

Approximation Algorithms

• yes/no problems => problems with numeric answers

• problem X: I (instance) => S(I) R (solution space)

• maximization problems: OPT(I) = max S(I)– max-value Knapsack, longest path

• minimization problems: OPT(I) = min S(I)– shortest TSP tour, smallest vertex cover

Approximation Algorithms

• Definition: – an algorithm A is an c-approximation for

a maximization problem X if for every instance I, A outputs A(I)S(I) such that OPT(I)/c ≤ A(I) ≤ OPT(I)

– an algorithm A is an c-approximation for a minimization problem X if for every instance I, A outputs A(I)S(I) such that OPT(I) ≤ A(I) ≤ cOPT(I)

• c: approximation ratio

How Do You Bound the Approximation Ratio?

• We do not know the value of OPT(I)• How can we prove that something

is within a constant factor from OPT(I)?• General idea (minimization version):

– find an easily computable lower bound on OPT(I)• i.e., OPT(I) ≥ X

– show that A(I) is within a constant factor of X• i.e., A(I) ≤ cX

Example: Vertex Cover

• Instance: Graph G=(V, E), target K• Question: is there a V’ V such that

– |V’|≤ K and for any (u, v) E either u V’ or v V’

• Optimization version: what is the size of the smallest vertex cover?

Vertex Cover: 2-Approximation

1. Start with V’ empty2. While there are unmarked edges left, repeat:

Pick an arbitrary edge e=(u, v) and mark itAdd u, v to V’Remove from G all unmarked edges incident to u or v

except (u, v)

Vertex Cover: Proof

• Claim 1: marked edges form a matching (do not share vertices)

• Claim 2: Given any matching M = {(u1, v1), ..., (uk, vk)}, any vertex cover (including OPT(I)) must contain at least one of ui, vi for each pair (ui, vi) in M

• V’ contains both– hence, V’ is at most twice as large as OPT(I)

Can We Do Better?

• Claim: unless P ≠ NP, any poly-time approximation algorithm for VC has approximation ratio at least 1+1/(n+1)

• Proof: suppose OPT(I) ≤ A(I) ≤ OPT(I)+OPT(I)/(n+1)– OPT(I) < n, so OPT(I)/(n+1) < 1– A(I), OPT(I) are both integer and differ by < 1– A(I) = OPT(I) and P = NP

Polynomial-time Approximation Scheme (PTAS)

• Definition (maximization version): A is a PTAS for a problem P if: – inputs: instance I of P, an error parameter – output: A(I, ) such that

(1-)OPT(I) ≤ A(I, ) ≤ OPT(I)– for every fixed , running time of A is poly(I)

• FPTAS: same, but running time of A is poly(I, 1/) for all

(n/2 is an FPTAS, n1/ is a PTAS, but not FPTAS)

Tool: Another Pseudopolynomial Algorithm for Knapsack

• Knapsack: • n items, each item has a weight wi and a value vi

• capacity W, target value V• poly (n, W, log V) algorithm: fill out table P(i, w):

maximal value of a subset of {1, ..., i} of weight w • Claim: there is an algorithm for Knapsack

that runs in time poly (n, V, log W)• Proof: fill out table W(i, v), i=1, ..., n, v=1, ..., V:

minimal weight of a subset of {1, ..., i} of value v

Tool: Another Pseudopolynomial Algorithm for Knapsack

• Claim: there is an algorithm for Knapsack that runs in time poly (n, V, log W)

• Proof: fill out table W(i, v), i=1, ..., n, v=1, ..., V: W(i, v): minimal weight of a subset of {1, ..., i} of value V

– W(1, v): +∞ if v ≠ v1 and w1 if v = v1

– W(i+1, v): min {W(i, v), wi+1 +W(i, v - vi+1)

• Check if W(n, v) ≤ W for some v ≥ V

Pseudopolynomial Algorithm => FPTAS

• Algorithm from previous slide has running time poly (n, V, log W)– polynomial if V = poly(n)– also polynomial if values are drawn from a

“small” set: • e.g., all values are of the form kX,

where k is at most poly(n); X can be huge

– idea: make the set of possible values small by rounding

FPTAS for Knapsack

• Knapsack: • n items, each item has a weight wi and a value vi

• capacity W, target value V• Parameter • Algorithm: set =vmax/n, and set v’i = max{k | k < vi}

• Let ki = v’i/; observe that ki ≤ n/(n/= vmax)• recall: W(i, v)=min weight of a subset of {1, ..., i} of value v• W’(i, k): min weight of a subset of {1, ..., i} of value k

• can be computed in the same way by DP

FPTAS for Knapsack: Bounding the Error

• W’(i, k): min weight of a subset of {1, ..., i} of value k • Compute W’(i, k) for i=1, ..., n, k=1, ..., n2/n2/≥

V) • Let V’ = max {k | W’(n, k) ≤ W}• How different is V’ from OPT?

– consider an opt solution i1, ...., it

– vi1 + ... + vit = OPT

– we have V’ ≥ v’i1 + ... + v’it

– also v’i1 ≥ vi1 - for all i1, ...., it

V’ ≥ OPT - n = OPT – vmax ≥ OPT – OPT ≥ OPT(1-)

More Complexity Theory

• Recall: 3-SAT is the question of whether a given Boolean formula is satisfiable

• Tautology: is a given formula a tautology, i.e., is it true for all values of variables?

• Does not seem to be in NP: what’s the witness?• Easy to prove that a formula is NOT a tautology:

exhibit a falsifying assignment• coNP: problems whose complements are in NP

More about coNP

• Complement of a problem: Q is a complement of P if a yes-instance of P is a no-instance of Q and vice versa

• How to prove that a problem is in coNP?– prove that its complement is in NP

• coNP-hard: as hard as any problem in coNP• How to prove that a problem is coNP-hard?

– prove that its complement is NP-hard

• It is believed that coNP ≠ P, coNP ≠ NP

Even Harder Problems (1/2)

• Minimal Circuit: given a Boolean circuit C with n inputs, is it minimal?– is it the case that for any circuit C’ with |C’|<|C|

there is an input x such that C’(x) ≠ C(x)?

• Does not seem to be in NP...• Complement: is a given Boolean circuit

non-minimal?– does not seem to be in NP either:

• even if you guess C’, need to show C’(x)=C(x) for all x

Even Harder Problems (2/2)

• Team Stability:– set of n agents N– any subset S of N can form a team and earn u(S)

• assume that u(S) is poly-time computable

• Question: can we split agents into teams S1, ..., Sk and share the earnings so that no group of agents wants to deviate?– is there a partition (S1, ..., Sk) and a vector (p1, ..., pn)

such that jSi pj = u(Si) for all i=1, ..., k and jS pj ≥ u(S) for all S N?

2 and 2

• Minimal Circuit: is it the case that for any circuit C’ with |C’|<|C|there exists an input x such that C’(x) ≠ C(x)?

• Team Stability: does there exist a partition (S1, ..., Sk) and a vector (p1, ..., pn) such that jSi pj = u(Si) for all i=1, ..., k and jS pj ≥ u(S) for all S N?

• 2: problems of the form xy P(x, y)

• 2: problems of the form xy P(x, y) (where P(x, y) is polynomial-time checkable)

Polynomial Hierarchy

– 2: problems of the form xy P(x, y)

– 2: problems of the form xy P(x, y)

– NP (=1): problems of the form xP(x)

– coNP (=1): problems of the form x P(x)

– can extend this further:• k: k alternations of quantifiers starting with

• k: k alternations of quantifiers starting with

– It is believed that all these classes are distinct– can show k k+1, k+1 and k k+1, k+1

PSPACE

• PSPACE: the class of problems you can solve using polynomial space

• P PSPACE: can’t use up more than poly space in poly time

• NP PSPACE: can check all candidates reusing the space

• 2 PSPACE: e.g., for Min Circuit, need– poly(n) bits to go over circuits– poly(n) bits to go over Boolean strings

More about PSPACE

• all of polynomial hierarchy is in PSPACE• EXPTIME: problems solvable in time 2poly(n)

– PSPACE EXPTIME

• are there PSPACE-complete problems?• Unbounded Quantifier Alternation: check if

x1y1x2y2....P(x1, y1, x2, y2, ....)is satisfiable

• Claim: Unbounded Quantifier Alteration is PSPACE-complete

Problems in PSPACE: Other Examples

• given a 2-player game, does the first player have a winning strategy?– does there exist a move for player 1 such that

for any move of player 2there exists a move of player 1 such that....

• Any deterministic 2-player game that is guaranteed to terminate is in PSPACE

• Some are PSPACE-complete: a version of GO

Counting Solutions

• Sometimes you want to know how many solutions are there– e.g., if there are many, you can sample

• #P: class of counting problems whose corresponding decision problems are in NP– #SAT: how many satisfying assignments

a given formula has• #P-completeness:

complete under counting reductions (preserve the number of solutions)– #SAT is #P-complete

Summary

• Lecture 1: – polynomial time– P vs. NP– NP-complete problems

• Lecture 2:– strong NP-hardness– pseudopolynomial algorithms– approximation algorithms– polynomial hierarchy, PSPACE, #P