CS 2430

Day 28

Announcements

• We will have class in ULR 111 on Monday

• Exam 2 next Friday (sample exam will be distributed next week)

Agenda (long term)

• Computational complexity

• Searching

– Linear search

– Binary search

• Sorting

Computational complexity and big O notation

Algorithm efficiency

• Need a way to determine how “efficient” an algorithm so we can compare algorithms

• Could try timings, but they can vary a lot depending on the input and/or hardware

• Another approach is to count the number of steps required to process N elements

Critical steps

• To get a feel for the behavior of the algorithm, we just count the dominant instructions

– for a sort, it would be the comparison of array elements

– for a search, it would be the comparison of an array element to the item being searched for

• The count will be some function of N, i.e., the number of items being processed

Big O

• The “Big O” of an algorithm is the order of the fastest growing term of the number of critical steps performed (as a function of N)

• Examples:– 2N / 5 + 3 = O(N)– 42 = 42N0 = O(1)– 100N2 + 7N / 2 + 15 = O(N2)

– Log N + 31 = O(Log2 N), where “Log2 N” is the base-2 logarithm of N

Big O examples

• Search an unsorted array?

O(N)

• Push onto a Stack?

O(1)

• Remove from a Bag?

O(N)

• Compute the union of two disjoint sets?

???

Matrix multiplication

public class Matrix

{

private double[][] data; // matrix entries

private int n; // dimension of matrix, assume square

. . .

public Matrix times(Matrix rhs)

{

Matrix lhs = this;

Matrix res = new Matrix(lhs.n); // creates empty Matrix

for (int i = 0; i < res.n; i++)

for (int j = 0; j < res.n; j++)

for (int k = 0; k < lhs.n; k++)

res.data[i][j] += lhs.data[i][k] * rhs.data[k][j];

return res;

}

}

Let N denote the matrix dimension, i.e., lhs.n

What is the big O of times()?

O(N3)

Do YOU think it’s possible to do better than O(N3)?

Yes!

Middle school matrix multiplication algorithm O(N3)

Strassen algorithm (1969) O(N2.807)

Coppersmith–Winograd algorithm (1987, 1990)

O(N2.376)

Williams algorithm (2011) O(N2.373)

<your name here> (20??) O(N2) – conjectured to be possible!

Bigger big O and beyond!

Agrawal–Kayal–Saxena primality test algorithm (2002)

O(N10) –The exponent continues to decrease

Traveling Salesman problem (NP-complete)

O(N!) – brute forceO(2N) – “Dynamic programming”

Halting problem (undecidable) Cannot be solved by any computer given any amount of time. No big O!Cannot even be solved by humans (in general)

Searching

Linear searchprivate static <E> int lsearch(Comparable<E> a[], int n, E x)

{

for (int i = 0; i < n; i++)

if (a[i].compareTo(x) == 0)

return i;

return -1;

}

Can put this method in any container class that needs it

What is the worst case big O?

O(N)

Linear search on average

• What is the average case efficiency of linear search?

• Given a random array that contains the target element X, how many comparisons needed?

• Answer = 1*Pr[X is in position 0] + 2*Pr[X is in position 1] + ∙ ∙ ∙ + N*Pr[X is in position N – 1] = ???

• Answer = O(N)

• How did we get this?

How to get the answer!

Answer = 1*Pr[X is in position 0] + 2*Pr[X is in position 1] + ∙ ∙ ∙ +

N*Pr[X is in position N – 1]

= 1 / N + 2 / N + ∙ ∙ ∙ + N / N

= (1 / N) * N*(N+1) / 2

= (N + 1) / 2

= O(N)

Can we do better if the array is sorted?

Yes!

Binary search

• Array MUST be in sorted order (either ascending or descending)

• Algorithm: how do you look up a word in the dictionary?

• At each iteration of the algorithm, disregard half of the input that is currently under consideration

• Extremely efficient!

Search for “17”

2 3 5 7 11 13 17 19 23 29 31 37 41 43 47

2 3 5 7 11 13 17

11 13 17

17

(0 + 14) / 2 == 7

(0 + 6) / 2 == 3

(4 + 6) / 2 == 5

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

0 1 2 3 4 5 6

4 5 6

6

Binary search algorithmprivate static <E> int bsearch(Comparable<E> a[], int n, E x)

{

int lo = 0, hi = n - 1;

while (lo <= hi)

{

int mid = (lo + hi) / 2;

int result = a[mid].compareTo(x);

if (result == 0)

return mid;

if (result > 0)

hi = mid - 1;

else

lo = mid + 1;

}

return -1;

}

Another example

private static <E> int bsearch(Comparable<E> a[], int n, E x)

{

int lo = 0, hi = n - 1;

while (lo <= hi)

{

int mid = (lo + hi) / 2;

int result = a[mid].compareTo(x);

if (result == 0)

return mid;

if (result > 0)

hi = mid - 1;

else

lo = mid + 1;

}

return -1;

}

►

►

►

►

►

►

►

►

►

►

►

►

►

2 3 5 7 11 13 17 19 23 29 31 37 41 43 47

lo

mid

hi

17 x

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

private static <E> int bsearch(Comparable<E> a[], int n, E x)

{

int lo = 0, hi = n - 1;

while (lo <= hi)

{

int mid = (lo + hi) / 2;

int result = a[mid].compareTo(x);

if (result == 0)

return mid;

if (result > 0)

hi = mid - 1;

else

lo = mid + 1;

}

return -1;

}

►

►

►

►

►

►

►

►

►

►

►

►

►

2 3 5 7 11 13 17 19 23 29 31 37 41 43 47

lo

mid

hi

35 x

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Binary search efficiency

Worst case scenario

• What if target item not present?

• How many iterations of the loop?

• Each time through the loop, roughly half the array elements are disregarded

Worst case big O

• How many times can you divide N by 2 before you get 1?

• Example: N = 100; 100 / 2 = 50; 50 / 2 = 25; 25 / 2 = 12; 12 / 2 = 6; 6 / 2 = 3; 3 / 2 = 1; Answer: 6 times

• In general, solve for r: N / 2r = 1

• We get: r = Log2 N (we’ll just write “Log N”)

• Final answer: O(Log N)

Any questions?

Review

• Linear search: O(N) (worst and average case)

• Binary search: O(Log N) (worst case)

• Binary search works with sorted input

• How to get the input sorted?

Next: sorting