1

Complexity

Lecture 17-19

Ref. Handout p 66 - 77

2

Measuring Program Performance

How can one algorithm be ‘better’ than another?

Time taken to execute ? Size of code ? Space used when running ? Scaling properties ?

3

Scaling

Amount of data

Time

4

Time and Counting

initialisefor (i = 1; i<=n; i++) {

do something}finish

Time needed:

t1set up loop t2each iteration t4

t5

The total ‘cost’ is t1+t2+n∗t4+t5 Instead of actual time, ti can be treated

as how many instructions needed

5

The computation cost can be estimatedas a function f(x) In the above example,

cost is f(x) = t1+t2+x∗t4+t5

= c + m∗x ( c = t1+t2+t5, m = t4 )

f(x) =c+mx

x (size of data)

c

m = slope

a linear function

6

The computation cost can be estimatedas a function f(x) – more examples

cost functions can be any type depending on the algorithm, e.g.f(x) = c ( c is a constant ), h(x) = log(x), or g(x) = x2

f(x) =c

x (size of data)

c

cost stays same

h(x) = log(x)

cost growsvery slowg(x) =x2

quadratic growth

7

The Time Complexity

A cost function for an algorithm indicates how computation time grows when the amount of data increases

A special term for it – TIME COMPLEXITYcomplexity = time complexity/space complexitySpace complexity studies the memory usage (but mostly we pay our attention to time complexity)

“Algorithm A’s time complexity is linear” = A’s computation time is proportional to the size of A’s input data

8

The Big-O Notation

How to compare (evaluate) two cost functions

Is n+100*n2 better than 1000*n2

It has been agreed to use a standard measurement – an order of magnitude notation – the Big-O notation

9

The Big-O Notation - definition

A function f(x) is said to be the big-O of a function g(x) if there is an integer N and a constant C such that

f(x) ≤ C∗g(x) for all x ≥ N

We write

f(x) = O(g(x))

10

The Big-O Notation - Examples

f(x) = 3∗xg(x) = xC = 3, N =0f(x) ≤ 3∗g(x)

f(x) = 4+7∗xg(x) = xC = 8, N =4f(x) ≤ 8∗g(x)

g(x) = x

f(x) = 3x = 3g(x)f(x) = 4+7x

g(x) = x

8*g(x)

3 x∗ = O(x) 4+7 x∗ = O(x)N

11

The Big-O Notation - Examples

f(x) = x+100∗x2

g(x) = x2

C = 101, N = 1f(x) ≤ 101∗g(x)

f(x) = 20+x+5∗x2+8∗x3

g(x) = x3

C = 15, N =20f(x) ≤ 9∗g(x)

when x > 1x+100∗x2 < x2+100∗x2

x2+100∗x2 = 101*g(x)

x+100∗x2= O(x2)

when x > 2020+x+5∗x2+8∗x3 <x3+x3+5∗x3+8x3

= 15∗x3 = 15*g(x)20+x+5∗x2+8∗x3= O(x3)

12

Power of n

A smaller power is always better eventually.for larger enough n

The biggest power counts most.for larger enough n

1000000000 * n < n2

10 *n4 + 23*n3 + 99*n2 + 200*n + 77 < 11*n4

13

Using Big-O Notation for classifyingalgorithms

23*n3 + 11

1000000000000000*n3 +100000000*n2 + ...

or even 100*n2 + n

All cost functions above is O(n3)

Performance is no worse than C*n3 eventually

14

Working Out Big-O

Ignore constants in multiplication - 10*n is O(n)

Leave products – n*log(n) is O( n*log(n) )

Use worst case in sums - n2 + 100*n + 2 is O(n2)

15

Memo for In-class test 17 [ /5]

questions my answers

correct answers

comments

1

2

3

4

5

16

Some Typical Algorithms - Search

Search an array of size n for a particular item

Sequential search

Time taken is O( )

Binary search

Time taken is O( )

17

Some Typical Algorithms - Sorting

Insertion sort Use 2 lists – unsorted and sorted

insertionSort(input_list) {

unsorted = input_list (assume size = n)

sorted = an empty list

loop from i =0 to n-1 do {

insert (unsorted[i], sorted) }

return sorted

}

Time taken is O( )

18

Some Typical Algorithms - Sorting

Bubble sort

Swap if xi < xi-1 for all pairs

Do this n times (n = length of the list)

x0 3

x1 5

x2 1

x3 2

x4 6

Time taken is O( )

19

Some Typical Algorithms - Sorting

Quick sort

quickSort(list) {

If List has zero or one number return list

Randomly pick a pivot xi to create two partitions larger than xi and smaller than xi

return quickSort(SmallerList)

+ xi + quickSort(LargerList)

}

// ‘+’ means appending two lists into one

Time taken is O( )

20

Worst Cases, Best Cases, and Average Cases

Complexity:performance on the average cases

Worst/Best cases:The worst (best) performance which an algorithm can get

E.g. 1 - bubble sort and insertion sort can get O(n) in the best case (when the list is already ordered)

E.g. 2 – quick sort can get O(n2) in the worst case (when partition doesn’t work well)

21

Sorting a sorted list

123456789

22

A Summary on Typical Algorithms

Sequential search O(n) Binary search (sorted list) O(log(n))

Bubble sort O(n2) best O(n) worst O(n2) Insertion sort O(n2) best O(n) worst O(n2)

Quick sort O(log(n)*n) best O(log(n)*n)

worst O(n2)

23

Can you order these functions ?smaller comes first

log(n)

log(n)*n n4/3

√n

n 2n

(log(n))2

n2

24

Memo for In-class test 18 [ /5]

questions my answers

correct answers

comments

1

2

3

4

5

25

Speeding up the Machine

can solve problem of size N in one day machine speed

up

Complexity

X 1000 X 1000000

O( log(n) ) N1000 N1000000

O( n ) 1000 * N 1000000 * N

O( n2 ) 32 * N 1000 * N

O( n3 ) 10 * N 100 * N

O( 2n ) N + 10 N + 20

size

26

Code Breaking

A number is the product of two primes

How to work out prime factors?

i.e. 35 = 7*5

5959 = 59*101

48560209712519 = 6850049 * 7089031

Complexity is O(10n) where n is the number of digits ( or O(s) where s is the number)

Typically n is about 150

27

Cost of Speeding Up

For each additional digit multiply time by 10( as the complexity is O(10n) )

For 10 extra digitsincrease time from 1 second to 317 years

For 18 extra digitincrease time from 1 second totwice the age of the universe

28

Where Different Formulas Appear

O(log(n)) find an item in an ordered collection

O( n ) look at every item

O( n2 ) for every item look at every other item

O( 2n ) look at every subset of the items

29

Do Big-O Times Matter?

Is n3 worse than 100000000000000000*n ?

Algorithm A is O(n2) for 99.999% of casesand

O(2n) for 0.001% cases

Algorithm B is always O(n10)

Which one is better?

30

Degree of Difficulty

log(n)

n*log(n)

nn, n (2^n)

n, n2, n3, ......

2n, 3n, ......

polynomial time

‘tractable’

exponential time

‘intractable’

31

A Simple O(2n) Problem

Given n objects of different weights,

split them into two equally heavy piles

(or say if this isn’t possible)

32

Hard Problem Make Easy

Magic Machine(for problem A)

Possible Datafor problem A

CorrectAnswer? Answer

checker

33

NP Problems

A problem is in the class NP

(Non-deterministic Polynomial Time)

if the checking requires polynomial time

i.e. O(nc) where C is a fixed number)

Note: problems not algorithms

34

Example

Given n objects of different weights,

split them into two equally heavy piles

{1,2,3,5,7,8}

Algorithm

1+2+3+7 = 8+5

Checkingok ✔

O(2n)

O(n)

35

In Previous Example ...

We know there are algorithms takes O(2n)

But ......

How do we know there isn’t a better algorithm which only takes polynomial time anyway?

36

Some Classes of Problems

P

NP

problems which canbe solved in polynomialtime using a magic box

problems which can be solved in polynomial time

37

Problems in NP but not in P

NPP

NPP?=

is a subset of

but

Most problems forwhich the best knownalgorithm is believed to be O(2n) are in NP

P

NP

38

The Hardest NP Problems

If a polynomial time algorithm is found for any problem in NP-Complete then every problem in NP can be solved in polynomial time.

i,e, P = NP

P

NP

NP-Complete

39

Examples of NP-Complete Problems

The travelling salesman problem Finding the shortest common

superstring Checking whether two finite automata

accept the same language Given three positive integers a, b, and

c, do there exist positive integers such that a*x2 + b*y2 = c

40

How to be Famous

Find a polynomial time algorithm for this problem

Given n objects of different weights,

split them into two equally heavy piles

(or say if this isn’t possible)

41

Memo for In-class test 19 [ /5]

questions my answers

correct answers

comments

1

2

3

4

5