Algorithms and Complexity - University of California, … Thinking Algorithms and Complexity Role of Algorithms in Modern World E-commerce (Amazon, Ebay) Network tra c (telecom billing,

Computational Thinking Algorithms and Complexity

Algorithms and Complexity

Cetin Kaya Kochttp://koclab.cs.ucsb.edu/teaching/cs192

[email protected]

Cetin Kaya Koc http://koclab.cs.ucsb.edu Fall 2016 1 / 56

http://koclab.cs.ucsb.edu/teaching/cs192

[email protected]

http://koclab.cs.ucsb.edu


Role of Algorithms in Modern World

E-commerce (Amazon, Ebay)

Network traffic (telecom billing, monitoring)

Database transactions (Sales, inventory)

Scientific measurements (astrophysics, geology)

Sensor networks and RFID tags

Bioinformatics (genome, protein bank)




Examples of Famous Algorithms

Geometric Constructions of Euclid

Newton’s root finding

Fast Fourier Transform

Compression (Huffman, Lempel-Ziv, GIF, MPEG)

DES, DH, RSA encryption

Simplex algorithm for linear programming

Shortest Path Algorithms (Dijkstra, Bellman-Ford)

Minimum Spanning Tree Algorithms (Prim, Kruskal)

Error correcting codes (CDs, DVDs)

TCP congestion control, IP routing

Pattern matching (Genomics)

Search Engines (Page Ranking)




A Real-World Problem

Communication in the Internet

Message (email, ftp) broken down into IP packets

Sender/receiver identified by IP address

The packets are routed through the Internet by special computerscalled Routers

Each packet is stamped with its destination address, but not the route

Because the Internet topology and network load is constantlychanging, routers must discover routes dynamically

What should the Routing Table look like?




IP Prefixes and Routing

Each router is really a switch: it receives packets at several inputports, and appropriately sends them out to output ports

Thus, for each packet, the router needs to transfer the packet to thatoutput port that gets it closer to its destination

Should each router keep a table: IP address x Output Port?

How big is this table?

When a link or router fails, how much information would need to bemodified?

A router typically forwards several million packets/sec!




Data Representation and Structures

The IP packet forwarding is a Data Representation and Structureproblem!

Efficiency, scalability is very important.

Similarly, how does Google find the documents matching your queryso fast?

Uses sophisticated algorithms to create index structures, which arejust data structures

Algorithms and data structures are ubiquitous

With the data glut created by the new technologies, the need toorganize, search, and update MASSIVE amounts of information FASTis more severe than ever before




Why Efficient Algorithms Matter

Suppose n = 106

A PC can read/process n records in 1 second

But if some algorithm does O(n2) computation, then it takes 1million seconds = 11 days!!!

100 City Traveling Salesman Problem

A supercomputer checking 100 billion tours/second still requires10,100 years!

Fast factoring algorithms can break the RSA

Algorithms research determines what is the safe key length




How to Measure Algorithm Performance

What metric should be used to judge algorithms?

Length of the program (lines of code)

Ease of programming (bugs, maintenance)

Memory required

Running time

Running time is the dominant standard.

Quantifiable and easy to compare

Often the critical bottleneck




Abstraction

An algorithm may run differently depending on:the hardware platform (PC, Cray, Sun)the programming language (C, Java, C++)the programmer (you, me, Bill Joy)

While different in detail, all hardware and prog models are equivalentin some sense: Turing machines

It suffices to count basic operations

Crude but valuable measure of algorithm’s performance as a functionof input size




Analysis of Algorithms

If there are several algorithms for solving the same computationalproblems, which algorithm shall we choose?

This is an important question in computer science

There may be several criteria for specific computational problems:

the fastestthe least amount of memorythe easiest one to write and maintain

A more general and useful approach has been to choose the algorithmthat will produce the output the fastest ← most efficient!

This brings us to the subject of running time analysis of algorithms




Analysis of Algorithms

The running time analysis of algorithms involves the counting andfinding the final sum T (n) of the “atomic” operations required by thealgorithm with input size n

The definition of an atomic operation depends on what the algorithmdoes in principle, and what we need to count in order to understandand quantify its running time

It is also assumed that the total time (in seconds, hours, etc) wouldbe proportional to T (n)




Search Algorithm

For example, for a searching algorithm, we will need to count thenumber of “comparisons”, because that what searching does: findingthe index of x in a list L

At the it step, we compare x to the ith element of the list:if x = L(i), we found x and its index is i

Search of x = 15 in List L below:

0 1 2 3 4 5 6 i23 11 41 15 17 35 20 L(i)

Input size n = 7, the number of elements of the list




Atomic Operations

The atomic operation will be different for other algorithms, forexample, we will have to count

The number of “compare” operations in a search algorithm

The number of “swap” operations in a sorting algorithm

The number of single-word addition and multiplication operations in amulti-word multiplication algorithm

The number of additions and multiplications in an algorithm thatcomputes the product of two n × n matrices




Worst, Average, and Best Case Analysis

When we count the number of operations, we will need to consider“the worst case” or “the average case” or “the best case” situations,in terms the input configuration, values or other factors

Therefore, we would be performing the the worst case, the averagecase or the best case analysis of the algorithm we are considering

In most situations, we will have to consider the worst case analysis inorder to make a fair assessment of the algorithm, and to compare itto other algorithms computing the same output




Binary Exponentiation Algorithm

We will describe an algorithm that computes xe given the input x(which can be an integer, or a real number, or even a matrix) and theexponent e (which is a positive integer)

ei be the ith bit of the exponent for 0 ≤ i ≤ n − 1

In other words e = (en−1en−2 · · · e1e0)

Example: e = 143 = (10001111) and n = 8

e7 e6 e5 e4 e3 e2 e1 e01 0 0 0 1 1 1 1

What is the input? x and e

What is the input size?

Answer: The number of bits in e





The input size of the algorithm is n which is the number of bits in e

Step 1: y = 1

Step 2: for i=n-1 down to 0

Step 3: y = y * y

Step 4: if (e_i=1) y = y * x

Step 5: return y





When e = (10001111), we start with y = 1 and compute:

e

1: y = 1 * 1 = 1 ... y = 1 * x = x

0: y = y * y = x^2 ...

0: y = y * y = x^4 ...

0: y = y * y = x^8 ...

1: y = y * y = x^16 ... y = x^16 * x = x^17

1: y = y * y = x^34 ... y = x^34 * x = x^35

1: y = y * y = x^70 ... y = x^70 * x = x^71

1: y = y * y = x^142 ... y = x^142 * x = x^143




Analysis of Binary Exponentiation Algorithm

Obviously the atomic operation in this algorithm is the multiplicationor squaring operation: y = y * y and y = y * x

We count the number of multiplications (assuming squaring takes thesame time)





The running time of the binary exponentiation algorithm T (n) is thenumber of multiplications, which depends on the bit configuration ofthe exponent

The exponent e could be (100 · · · 0) or (111 · · · 1) or (1010 · · · 10),giving us different T (n) values

More 1s in e imply more multiplications in Step 4





The best case analysis: there exists only a single 1 in the binaryexpansion of the exponent: e = (1000 · · · 0)

In Step 3, we will have n multiplications, and in Step 4, we will haveonly 1, therefore, Tbest(n) = n + 1





The worst case analysis: there exists only n 1s in the binary expansionof the exponent: e = (1111 · · · 1)

In Step 3, we will have n multiplications, and in Step 4, we will alsohave n multiplications, therefore, Tworst(n) = 2n





The average case analysis: Assuming each exponent bit is equal 1 or0 with equal probability, there will be about n/2 1s and n/2 0s in thebinary expansion of the exponent

In Step 3, we will have n multiplications, and in Step 4, we will haven/2 multiplications, therefore, Taverage(n) = 1.5n





Therefore, the binary exponentiation algorithm requires

Tbest(n) = n + 1Tworst(n) = 2n

Taverage(n) = 1.5n

multiplications, depending on whether the Hamming weight of e is 1or n or n/2




Analysis of Sequential Search Algorithm

On the other hand the search algorithm requires

Tbest(n) = 1Tworst(n) = n

Taverage(n) = n2

comparisons, depending on whether x is found in the first cell, in thelast cell, or in a cell that is about in the middle of the array




Comparing Algorithms

By counting the number of operations T (n) in the analysis of analgorithm, we stay away from (useless) discussions and investigationsof the details of the computer we run our program on, such as clockspeed, the total time in microseconds of an atomic operation

Our objective was to choose between two algorithms that solve thesame computational problem

We simply compare their running times: T1(n) and T2(n)

If T1(n) ≤ T2(n) for all n after a certain n > n0, then we must selectAlgorithm 1

For example, given T1(n) = 4n2 and T2(n) = 40n2, we should preferAlgorithm 1





Given T1(n) = n2 + n and T2(n) = 1000n, which algorithm?

Why?

Clearly: T1(10) = 110 < T2(10) = 10, 000

However, as n grows, T1(n) gets larger faster

After a certain n > n0, we have T1(n) > T2(n)

For example, T1(1000) = 1, 001, 000 > T2(1000) = 1, 000, 000

For every n > 1000, we have T1(n) > T2(n)





Algorithm 1 asymptotically takes more time complete

Algorithm 2 should be preferred

Question: Would you prefer Algorithm 1 for n < 1000 ?




Big-O Notation

Big-O notation allows to compare running times of algorithms tomake better decisions

This is a step separate from the counting operation, it helps us toquickly ascertain the performance of the algorithm we have analyzed

We say T (n) = O(f (n)) if we can find C and n > n0 such thatT (n) ≤ C · f (n) for n > n0

For example, if T (n) = 2n, we have T (n) = O(n)




Big-O Notation

In general, if

T (n) = aknk + ak−1n

k−1 + · · ·+ a1n + a0

then T (n) = O(nk), i.e., the largest power in the polynomialexpression

For example:

n2 + 1000n = O(n2)1000n2 + n = O(n2)

n3

100 + 1000n2 + 10000n = O(n3)

The asymptotic analysis of algorithms always assumes n growsunboundedly in order to find the algorithm that requires the leastamount of operations as n→∞




Examples of Running Times: Integers

Addition or multiplication of two n-bit integers

The atomic operation: The addition of two 1-bit numbers

ci+1 ci ci−1ai ai−1bi bi−1

ci+1 si ci si−1

Addition of two n-bit integers: O(n) time

Multiplication of two n-bit integers: O(n2) time

Karatsuba multiplication algorithm: O(nlog2 3) = O(n1.58) time

The FFT (fastest known) integer multiplication algorithm:O(n log n log log n) time




Examples of Running Times: Matrices

Addition or multiplication of two n × n matrices

The matrix entries (numbers) are integer, real or complex

The atomic operation: The addition of two 1-word numbers

c = a + b

Addition of two n × n matrices: O(n2) time

Multiplication of two n × n matrices: O(n3) time

Strassen multiplication algorithm: O(nlog2 7) = O(n2.81) time




Examples of Running Times: Sorting and Searching

Searching an element in a set of n elements

Sorting a set of n elements

The atomic operation: A comparison

Linear search in an unsorted array: O(n) time

Binary search in a sorted array: O(log n) time

Bubble Sort, Insertion Sort: O(n2) time

Merge Sort, Quick Sort: O(n log n)




Examples of Running Times: Graphs

Graph search algorithms: n is the number of nodes and e is thenumber of edges

The atomic operation: A comparison or marking

Prim’s algorithm (using adjacency matrix): O(n2) time

Prim’s algorithm (using adjacency list): O(e log n) time

Dijkstra algorithm (finding the shortest path between two nodes):O(e + n) time




Fast Algorithms

Some algorithms are super fast

For example, finding the smallest element in a SORTED list

It is first element in the list

Time complexity: Constant, O(1)

Some other algorithms are very fast

For example, finding a particular element in a SORTED list

The binary search: O(log n)

Some other algorithms are fast

For example, finding a particular element in an UNSORTED list

The linear search: O(n)




Polynomial-Time Algorithms

Many algorithms require polynomial time: (nk) where k is smallpositive real number

Multiplication of integers: k = 2

Karatsuba multiplication of integers: k = 1.58

Addition of matrices: k = 2

Multiplication of matrices: k = 3

Strassen multiplication of matrices: k = 2.81




Exponential-Time Algorithms

Some algorithms require exponential time

Discrete Logarithm Problem (DLP): Given y , p, g , find x in

y = g x (mod p)

such that p is prime and g is a primitive root

Integer Factorization Problem (IFP): Given n, find p or q in

n = p · q

such that p and q are primes




Complexity of Solving DLP

The DLP problem can be solved with p − 1 exponentiations:

for i in range(1,p-1):

if( (g**i)%p == y ):

print("Found it!, x = ",i)

If p is a k-bit integer, the complexity of a single exponentiation isO(k) multiplications (of k-bit numbers)

The complexity of a single exponentiation is O(k3) bit operations

Since we perform p − 2 < 2k exponentiations to find x , thecomplexity of solving DLP is O(k3 · 2k)




Complexity of Solving IFP

Note that either p <√n or q <

√n

The IFP problem can be solved with m =√n divisions

for i in range(1,m):

if( (n/i) is integer):

print("Found it!, x = ",i)

If n is a k-bit integer, m would be k/2-bit integer

The complexity of a single division is O(k2) bit operations

The complexity of solving IFP is O(k2 · 2k/2) since we performm < 2k/2 divisions to find p




Polynomial vs Exponential Times

Polynomial-time algorithms are of complexity O(nk)

Exponential-time algorithms are of complexity O(2n)

There seems to be a strong separation between them

Assume the computer can perform 1b operations per second:




NP-Hard Problems

Many exponential time problems seem to have a common property

Finding a solution is hard: it requires exponential time

However, if someone claims to know an answer, it can be checkedvery quickly (definitely not exponential time)

DLP: if someone claims to know the answer x in

y = g x (mod p)

we compute y ′ = g x (mod p) and see if it is correct y ′?= y




NP-Hard Problems

IFP: if someone claims to know the answer p in

n = p · q

we compute q′ = n/p and see if it is correct “is q′ integer”?

Definition: The set NP-hard

NP-hard contaims problems whose solutions require exponential time,but whose solutions can be verified in polynomial time

NP stands for nondeterministic polynomial

A nondeterministic computer “knows” the answer

The answer can be checked in deterministic polynomial time




Other Difficult Problems

Traveling Salesman Problem (TSP): Given a list of cities and thedistances between each pair of cities, what is the shortest possibleroute that visits each city exactly once and returns to the origin city?

20

40

60

80

100

20 40 60 80 100

c1

c7

c6

c5

c4

c3

c2

c1




Solving the TSP

We need to consider all permutations of the cities:c1 → c2 → c3 → · · ·c5 → c3 → c1 → · · ·c3 → c7 → c5 → · · ·For 7 cities, there are 6! = 720 permutations

For 10 cities, there are 9! = 362880 permutations


For 40 cities, there are

39! = 20397882081197443358640281739902897356800000000

permutations




Solving the TSP





39! = 20397882081197443358640281739902897356800000000

permutations




Solving the TSP





39! = 20397882081197443358640281739902897356800000000

permutations




Solving the TSP





39! = 20397882081197443358640281739902897356800000000

permutations




Complexity of the TSP

Since n! is O(nn), the complexity of the TSP seems to be anexponential function in terms of the number cities

A related problem, called the “decision TSP”: Given the edge costs(distances) and a number X , decide whether there is a round-triproute cheaper than X

The decision TSP belongs to a set of problems called NP-complete




NP-Complete Problems

The set NP-complete is a subset of NP

Therefore, we do not have a polynomial time algorithm for anyNP-complete problem either

However, if there is a solution, it can be verified in polynomial time

The set NP-complete has an additional and important property

These problems are very special in the sense that they are “reducible”to one another

A solution for one can be “encoded” or “transformed” to a solutionto another in polynomial time




P versus NP

P denotes the set of problems solvable in polynomial time

Is P = NP or P 6= NP?

So far, no one was able to invent a polynomial-time algorithm for anNP-complete problem, nor anyone was able to show that there cannotbe a polynomial time algorithm for an NP-complete problem




Arithmetic Model of Computer

In analyzing the running time of the algorithms, we constructed anabstract computer

Our abstract computer had strong resemblance to a real computer

It has a particular word length: 8, 16, 32 or 64 bits

It has an ALU capable of adding, subtracting, multiplying, orperforming a logical operation on two 1-word operands

This computer model was very suitable for arithmetic operations

However, we also had to consider bit-level operations: adding,subtracting, multiplying, or performing a logical operation on two1-bit operands




Turing Machine

If we want to study algorithms more abstractly, we need to consider acomputer that performs simple 1-bit operations and also has simpleprogramming capability

The English mathematician, cryptographer, and the “father ofcomputer science”, Alan Turing (1912-1954) envisioned such anabstract computer which we name the Turing Machine

Turing was considering a model that can answer some of thequestions the German mathematician David Hilbert formulated

Three of the questions were related to logical foundations ofmathematics




Hilberts’ Questions




Entscheidung Problem

The third question is named the decision problem, or in German, theEntscheidungproblem

Hilbert believed that the answer of all 3 problems was yes

However, Kurt Godel in 1931 proved that mathematics must beincomplete

There are statements that can neither be proved nor disprovedstarting from a set of axioms

Therefore, Godel gave negative answers to first two questions

The third question was to be answered by Turing




Turing Machine

Particularly, Turing asked “Was there a definite method, a mechanicalprocess which could be applied to a mathematical statement, andwhich would come up with the answer as to whether it was provable?”

By using the phrase“mechanical process” he meant nothing morethan an algorithm, a set of detailed instructions that leads to thesolution of a problem

He equated the concept of “computability” with the ability of a verysimple machine to perform a computation

His idea was to imagine a machine that worked like a human“computer” who just had to follow a set of rules




Turing Machine

Turing idealized his computer as using a long strip of paper or “tape”on which to do the calculations

The tape is broken up into square “boxes” in each of which it canread or write a symbol

With this simple machine, Turing was able to define what was meantby “computability”

If one can use finite amount of tape and finite steps to perform acomputation (to compute an object, a number, a result), then thatobject, number, result is computable




Computability

Turing machine is the basic tool to prove computability

Think of this way: When we say, “is√

2 computable” what we areasking whether there is an algorithm for computing the digits(decimal, binary, hexadecimal) of

√2?

Similarly, “is π computable”?

Even though both numbers infinitely many digits (irrational), thereare effective algorithms for computing their digits

π and√

2 are computable

Are there uncomputable things? :)

Are there unsolvable problems on a computer?




3x + 1 Problem

There is a simple-to-describe but unsolved problem in mathematics

Suppose we have a program that takes a number as input and eitherdivides it by two, if the number is even, or triples it and adds one, if itis odd

The program then takes this new number as input and repeats theprocess

When the output is one, the program halts

Can we be sure that this will happen?

This is known as the 3x + 1 problem




3x + 1 Problem

An Example: Start with 7: 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5,16, 8, 4, 2, 1 ← Terminates!

With other values of starting numbers we find that the programsometimes terminates but for some starting values it just keeps ongenerating numbers with no repeating pattern until we decide wehave seen enough

This is one specific type of the “halting problem”: a program forwhich we cannot determine whether it terminates or not

More generally, the halting problem is concerned with the terminationof any program for any input




Halting Problem

Halting problem is uncomputable!

Proof:



Documents

Algorithms and Complexity - University of California, … Thinking Algorithms and Complexity Role of Algorithms in Modern World E-commerce (Amazon, Ebay) Network tra c (telecom billing,