1

Lower BoundsLower Bounds

Lower boundLower bound: an estimate on a minimum amount of work : an estimate on a minimum amount of work needed to solve a given problemneeded to solve a given problem

Examples:Examples: number of comparisons needed to find the largest element number of comparisons needed to find the largest element

in a set of in a set of nn numbers numbers number of comparisons needed to sort an array of size number of comparisons needed to sort an array of size nn number of comparisons necessary for searching in a sorted number of comparisons necessary for searching in a sorted

arrayarray number of multiplications needed to multiply two number of multiplications needed to multiply two nn-by--by-n n

matrices matrices

2

Lower Bounds (cont.)Lower Bounds (cont.)

Lower bound can beLower bound can be• an exact count an exact count • an efficiency class (an efficiency class ())

TightTight lower bound: there exists an algorithm with the same lower bound: there exists an algorithm with the same efficiency as the lower boundefficiency as the lower bound

ProblemProblem Lower bound Lower bound TightnessTightness sortingsorting ((nnloglog n n) yes) yes

searching in a sorted arraysearching in a sorted array (log(log n n) yes) yes

element uniqueness element uniqueness ((nnloglog n n) yes) yes

nn-digit integer multiplication -digit integer multiplication ((nn) unknown) unknown

multiplication of multiplication of nn-by--by-n n matrices matrices ((nn22) unknown) unknown

3

Methods for Establishing Lower BoundsMethods for Establishing Lower Bounds

trivial lower boundstrivial lower bounds

information-theoretic arguments (decision trees)information-theoretic arguments (decision trees)

adversary argumentsadversary arguments

problem reductionproblem reduction

4

Trivial Lower BoundsTrivial Lower Bounds

Trivial lower boundsTrivial lower bounds: based on counting the number of items : based on counting the number of items that must be processed in input and generated as outputthat must be processed in input and generated as output

ExamplesExamples finding max elementfinding max element

polynomial evaluationpolynomial evaluation

sortingsorting

element uniquenesselement uniqueness

Hamiltonian circuit existenceHamiltonian circuit existence

Conclusions Conclusions may and may not be usefulmay and may not be useful

be careful in deciding how many elements be careful in deciding how many elements mustmust be processed be processed

5

Decision TreesDecision Trees

Decision treeDecision tree — a convenient model of algorithms involving — a convenient model of algorithms involving comparisons in which:comparisons in which: internal nodes represent comparisonsinternal nodes represent comparisons leaves represent outcomes leaves represent outcomes

Decision tree for 3-element insertion sortDecision tree for 3-element insertion sort

a < b

b < c a < cyes

yes no

noyesno

a < c b < c

a < b < c

c < a < b

b < a < c

b < c < a

no yes

abc

abc bac

bcaacb

yes

a < c < b c < b < a

no

6

Decision Trees and Sorting AlgorithmsDecision Trees and Sorting Algorithms

Any comparison-based sorting algorithm can be represented Any comparison-based sorting algorithm can be represented by a decision treeby a decision tree

Number of leaves (outcomes) Number of leaves (outcomes) nn!!

Height of binary tree with Height of binary tree with nn! leaves ! leaves loglog22nn!!

Minimum number of comparisons in the worst case Minimum number of comparisons in the worst case loglog22nn!! for any comparison-based sorting algorithmfor any comparison-based sorting algorithm

loglog22nn!! n n loglog22nn

This lower bound is tight (mergesort)This lower bound is tight (mergesort)

7

Adversary ArgumentsAdversary Arguments

Adversary argumentAdversary argument: a method of proving a lower bound by : a method of proving a lower bound by playing role of adversary that makes algorithm work the hardestplaying role of adversary that makes algorithm work the hardestby adjusting input by adjusting input

Example 1: “Guessing” a number between 1 and Example 1: “Guessing” a number between 1 and n n with yes/nowith yes/no questionsquestionsAdversary: Puts the number in a larger of the two subsetsAdversary: Puts the number in a larger of the two subsets

generated by last question generated by last question

Example 2: Merging two sorted lists of size Example 2: Merging two sorted lists of size nn

aa11 < < aa22 < … < < … < aann and and bb11 < < bb22 < … < < … < bbnn

Adversary: Adversary: aai i < b< bjj iff iff ii < < jj

Output Output bb11 < < aa11 < < bb22 < < aa22 < … < < … < bbnn < < aann requires 2 requires 2nn-1 comparisons-1 comparisons

of adjacent elementsof adjacent elements

8

Lower Bounds by Problem ReductionLower Bounds by Problem Reduction

Idea: If problem Idea: If problem P P is at least as hard as problem is at least as hard as problem QQ, then a lower, then a lower

bound for bound for Q Q is also a lower bound for is also a lower bound for P. P.

Hence, find problem Hence, find problem Q Q with a known lower boundwith a known lower bound that canthat can be reduced to problem be reduced to problem PP in question. in question.

Example: Example: PP is finding MST for is finding MST for nn points in Cartesian plane points in Cartesian plane Q Q is element uniqueness problem (known to be in is element uniqueness problem (known to be in ((nnloglognn))))

9

Classifying Problem ComplexityClassifying Problem Complexity

Is the problemIs the problem tractabletractable,, i.e., is there a polynomial-time (O(i.e., is there a polynomial-time (O(pp((nn)) )) algorithm that solves it?algorithm that solves it?

Possible answers:Possible answers: yes (give examples)yes (give examples)

nono

• because it’s been proved that no algorithm exists at all because it’s been proved that no algorithm exists at all (e.g., Turing’s (e.g., Turing’s halting problemhalting problem))

• because it’s been be proved that any algorithm takes because it’s been be proved that any algorithm takes exponential timeexponential time

unknownunknown

10

Problem Types: Optimization and DecisionProblem Types: Optimization and Decision

Optimization problemOptimization problem: find a solution that maximizes or : find a solution that maximizes or minimizes some objective functionminimizes some objective function

Decision problemDecision problem: answer yes/no to a question: answer yes/no to a question

Many problems have decision and optimization versions.Many problems have decision and optimization versions.

E.g.: traveling salesman problemE.g.: traveling salesman problem optimizationoptimization: find Hamiltonian cycle of minimum length: find Hamiltonian cycle of minimum length decisiondecision: find Hamiltonian cycle of length : find Hamiltonian cycle of length mm

Decision problems are more convenient for formal investigation Decision problems are more convenient for formal investigation of their complexity.of their complexity.

11

Class Class PP

PP: the class of decision problems that are solvable in O(: the class of decision problems that are solvable in O(pp((nn)) )) time, where time, where pp((nn) is a polynomial of problem’s input size ) is a polynomial of problem’s input size nn

Examples:Examples: searchingsearching

element uniquenesselement uniqueness

graph connectivity graph connectivity

graph acyclicitygraph acyclicity

primality testing (finally proved in 2002)primality testing (finally proved in 2002)

12

Class Class NPNP

NPNP ( (nondeterministic polynomialnondeterministic polynomial): class of decision problems ): class of decision problems whose proposed solutions can be verified in polynomial time whose proposed solutions can be verified in polynomial time = solvable by a = solvable by a nondeterministicnondeterministic polynomial algorithmpolynomial algorithm

A A nondeterministic polynomial algorithmnondeterministic polynomial algorithm is an abstract two- is an abstract two-stage procedure that:stage procedure that:

generates a random string purported to solve the problemgenerates a random string purported to solve the problem checks whether this solution is correct in polynomial timechecks whether this solution is correct in polynomial timeBy definition, it solves the problem if it’s capable of generating By definition, it solves the problem if it’s capable of generating

and verifying a solution on one of its tries and verifying a solution on one of its tries

Why this definition?Why this definition? led to development of the rich theory called “computational led to development of the rich theory called “computational

complexity”complexity”

13

Example: CNF satisfiabilityExample: CNF satisfiability

Problem: Is a boolean expression in its conjunctive normal Problem: Is a boolean expression in its conjunctive normal form (CNF) satisfiable, i.e., are there values of its form (CNF) satisfiable, i.e., are there values of its variables that makes it true?variables that makes it true?

This problem is in This problem is in NPNP. Nondeterministic algorithm:. Nondeterministic algorithm: Guess truth assignmentGuess truth assignment Substitute the values into the CNF formula to see if it Substitute the values into the CNF formula to see if it

evaluates to trueevaluates to true

Example: Example: ((A | A | ¬B ¬B | | ¬C¬C)) & & ((AA | | BB)) & & ((¬¬B | B | ¬D |¬D | EE)) && ((¬D | ¬E¬D | ¬E))Truth assignments:Truth assignments:

A B C D EA B C D E0 0 0 0 00 0 0 0 0

. . .. . .1 1 1 1 11 1 1 1 1

Checking phase: Checking phase: O(O(nn))

14

What problems are in What problems are in NPNP??

Hamiltonian circuit existence Hamiltonian circuit existence Partition problem: Is it possible to partition a set of Partition problem: Is it possible to partition a set of nn

integers into two disjoint subsets with the same sum?integers into two disjoint subsets with the same sum? Decision versions of TSP, knapsack problem, graph Decision versions of TSP, knapsack problem, graph

coloring, and many other combinatorial optimization coloring, and many other combinatorial optimization problems. (Few exceptions include: MST, shortest paths)problems. (Few exceptions include: MST, shortest paths)

All the problems in All the problems in PP can also be solved in this manner (no can also be solved in this manner (no guessing is necessary), so we have: guessing is necessary), so we have:

PP NPNP

Big question: Big question: PP = = NP NP ??

15

NPNP-Complete Problems-Complete Problems

A decision problem A decision problem DD is is NPNP-complete-complete if it’s as hard as any if it’s as hard as any

problem in problem in NPNP, i.e.,, i.e., D D is in is in NPNP every problem in every problem in NPNP is polynomial-time reducible to is polynomial-time reducible to DD

Cook’s theorem (1971): CNF-sat is Cook’s theorem (1971): CNF-sat is NPNP-complete-complete

NP-completeproblem

NP problems

16

NPNP-Complete Problems (cont.)-Complete Problems (cont.)

Other Other NPNP-complete problems obtained through polynomial--complete problems obtained through polynomial-

time reductions from a known time reductions from a known NPNP-complete problem-complete problem

Examples: TSP, knapsack, partition, graph-coloring andExamples: TSP, knapsack, partition, graph-coloring and hundreds of other problems of combinatorial nature hundreds of other problems of combinatorial nature

knownNP-complete

problem

NP problems

candidate for NP -

completeness

17

P P = = NP NP ?? Dilemma RevisitedDilemma Revisited

P P = = NP NP would imply that every problem in would imply that every problem in NP, NP, including all including all NP-NP-complete problems,complete problems, could be solved in polynomial timecould be solved in polynomial time

If a polynomial-time algorithm for just one If a polynomial-time algorithm for just one NP-NP-complete complete problem is discovered, then every problem in problem is discovered, then every problem in NP NP can be can be solved in polynomial time, i.e., solved in polynomial time, i.e., P P = = NPNP

Most but not all researchers believe that Most but not all researchers believe that P P NP NP , i.e. , i.e. P P is a is a proper subset of proper subset of NPNP

NP-completeproblem

NP problems