Lecture 2Computational Complexity

Time Complexity

Space Complexity

In computer science, the time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as a function of the length of the string representing the input[1]:226.

The Importance of Analyzing the Running Time of an Algorithm

An example to illustrate the importance of analyzing the

running time of an algorithmProblem: Given an array of n elements A[1..n], sort the entries in A in non-decreasing order.

Assumption: Each element comparison takes 10-6 seconds on some computing machine

Conclusion: Time is undoubtedly an extremely precious resource to be investigated in the analysis of algorithms.

Question: How to Analysis Running Time?

( 1)

2

n n

log 1n n n

algorithm # of element comparisons

n=128

=27

n=1,048,567

=220

Selection Sort

Merge Sort 610 (128 7 128 1)

0.0008 seconds

610 (128 127) 2

0.008 seconds

6 20 2010 (2 20 2 1)

20 seconds

6 20 2010 (2 (2 1)) 2

6.4 days

Running Time

How to measure the efficiency of an algorithm from the point of view of time?

The running time usually increases with the growing of input size. So, the running time of an algorithm is usually described as a function of input size.1. What is the measure of time?

2. How to define the value of the function for input size n ? If the running time is only dependent on input size? If not only dependent on input size, but also on input?

Is actual (exact) running time a good measure? The answer is No. Why? Actual time is determined by not only the algorithm, but also many other

factors; The measure should be machine or technology independent; Our estimates of times are relative as opposed to absolute;

Time complexityIn computer science, the time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as a function of the length of the string representing the input[1]:226.

• Time complexity is commonly estimated by counting the number of elementary operations performed by the algorithm, where an elementary operation takes a fixed amount of time to perform.

Running Time (continued)

Notice! Different models assume different elementary steps!

Examples of usually used elementary operations Arithmetic operations: addition, subtraction, multiplication and division. Comparisons and logical operations. Assignments, including assignments of pointers.

Example

! For some algorithm, selection sort for example,

its running time is only dependent on input size.

For any input with input size n, the running time of selection sort algorithm is always cn(n-1)/2 + bn + a for some constants a, b, c.

Our main concern is not in small input instances but the behavior of the algorithms under investigation on large input instances,

especially with the size of the input in the limit. In this case, we are interested in the order of growth of the running time function.

Example

1

( ) cost times

1 for 2 to length[ ]

2 do [ ]

INSERTION SORT A

j A c n

key A j

2 1

3 Insert [ ] into the sorted

sequence [1.. 1] 0 1

4 1

c n

A j

A j n

i j

4

5 2

6 2

1

5 while 0 and [ ]

6 do [ 1] [ ] ( 1)

n

jj

n

jj

c n

i A i key c t

A i A i c t

7 2

8

7 1 ( 1)

8 [ 1] 1

n

jji i c t

A i key c n

* stands for the number of times the while loop test in line 5 is executed for that value of jt j

But for many algorithms, running time is dependent also on input structure!

Example (cont.)

)1()1()1()1()1()( 82

72

62

5421

nctctctcncncncnfn

jj

n

jj

n

jj

Then

It is tedious! And tj depends on different input!e.g., if A is initially in sorted order, then for each j=2, …, n, the while loop test in line 5 is only executed once, thus,tj=1 for j=2, …,n. And we have:

ban

ccccnccccc

ncncncncncnf

)()(

)1()1()1()1()(

854285421

85421

Which is a linear function, and the order of growth of the function isn.

Example (cont.)If the array is initially in decreasing order.

Then for each j=2, …,n, the while loop test is executed j times, thus tj=j for j=2, …,n. Thus, we have

1 2 4 5 6 7 82 2 2

1 2 4 5 6 7 82 2

1 2 4 8 2 4 8 5 6 7

25 6 7

( ) ( 1) ( 1) ( 1) ( 1) ( 1)

( 1) ( 1) ( ) ( 1) ( 1)

( 2)( 1) ( 1)( ) ( ) ( ) ( )( )

2 2

( )2 2 2

n n n

j j jj j j

n n

j j

f n c n c n c n c t c t c t c n

c n c n c n c j c c j c n

n n n nc c c c n c c c c c c

c c cn

5 6 71 2 4 8 2 4 5 8

2

( ) ( )2 2 2

c c cc c c c n c c c c

an bn c

Which is a quadratic function, the order of growth of which is n2

Asymptotic notionsHow to define running time in function of input size when running time is dependent also on input?

For simplicity, we need asymptotic notions!Our main concern is not in small input instances but the behavior of the algorithms under investigation on large input instances, especially with the size of the input in the limit. In this case, we are interested in the order of growth of the running time function.

Asymptotic Notations

e.g., the running time of INSERTION-SORT falls between a1n+b1 and a2n2+b2n+c2, and neither of them can represent the whole case. By using the asymptotic notations, the running time of INSERTION-SORT can be simplified.

In this course, we use three notations: O(.) : “Big-Oh” – the most usedΩ(.) : “Big omega”Θ(.) : “Big theta”

O-notation

Informal definition of O(.): If an algorithm’s running time t(n) is bounded above by a function g(n),

to within a constant multiple c, for n n0, we say that the running time of the algorithm is O(g(n)), and the algorithm is an O(g(n))-algorithm.

Don’t care

n0

c·g(n)

t(n)

N

Time

Obviously, O-notation is used to bound the worst-case running time of an algorithm

O-notation

Formal definition of O(.): A function t(n) is said to be in O(g(n)), if there exist some c > 0, n0 > 0,

such that 0 ≤ t(n) ≤ cg(n), for all n ≥ n0. Sometimes, also denote as t(n) = O(g(n)).

Don’t care

n0

c·g(n)

t(n)

N

Time

Ω-notation

Informal definition of Ω(.) : If an algorithm’s running time t(n) is bounded below by a function g(n),

to within a constant multiple c, for n n0, we say that the running time of the algorithm is Ω(g(n)), and the algorithm is called an Ω(g(n))-algorithm.

Don’t care

n0

t(n)

N

Time

c·g(n)

Ω-notation

Formal definition of Ω(.) : A function t(n) is said to be in Ω(g(n)), if there exist some c > 0, n0 > 0,

such that t(n) ≥ cg(n) ≥0, for all n ≥ n0. Sometimes, also denote as t(n) = Ω(g(n)).

Don’t care

n0

t(n)

N

Time

c·g(n)

Obviously, Ω -notation is used to bound the best-case running time of an algorithm

Θ-notation

Informal definition of Θ(.) : If an algorithm’s running time t(n) is bounded above and below by a

function g(n), to within a constant multiple c1 and c2, for n n0, we say that the running time of the algorithm is Θ(g(n)), and the algorithm is called a Θ(g(n))-algorithm, or of order Θ(g(n)).

Don’t care

n0

t(n)

N

Time

c2·g(n)

c1·g(n)

Θ-notation

Formal definition of Θ(.) : A function t(n) is said to be in Θ(g(n)) if there exist c1 > 0, c2 > 0,

and n0 > 0, such that 0 ≤ c2g(n) ≤ t(n) ≤ c1g(n), for all n ≥ n0.

Sometimes, also denote as t(n) = Θ(g(n)).

Don’t care

n0

t(n)

N

Time

c2·g(n)

c1·g(n)

g(n) is an asymptotically tight bound for t(n)

Example

The running time of INSERTION-SORT is O(n2), and (n), which means the running time of every input of size n for n n0 is up-bounded by a constant times n2, and lower-bounded by a constant times n when n is sufficiently large.

Notice that for some algorithm, INSERTION-SORT for example, there does not exist a function g(n) such that the running time of the algorithm is (g(n)).

Illustration of some typical asymptotic running time functions

We can see:The linear algorithm is obviously slower than the quadratic one and faster than logarithmic one, etc..

Three Types of Analysis

Given input size, the running time may vary on different input instance, e.g., the running time of INSERTION-SORT falls between linear and quadratic functions. Can we give an exact order?

To have an overall grading on the performance of an algorithm, we consider:

Best-Case Analysis: Too optimistic

Average-Case Analysis: Too difficult, e.g. the difficulty to define “average case”, the difficulty related with mathematics. And most time average-case running time is in the same order as worst-case running time.

Worst-Case Analysis: Very useful and practical. We will adopt this approach.

Example of Three Types of Analysis

1

( ) cost times

1 for 2 to length[ ]

2 do [ ]

INSERTION SORT A

j A c n

key A j

2 1

3 Insert [ ] into the sorted

sequence [1.. 1] 0 1

4 1

c n

A j

A j n

i j

4

5 2

6 2

1

5 while 0 and [ ]

6 do [ 1] [ ] ( 1)

n

jj

n

jj

c n

i A j key c t

A i A i c t

7 2

8

7 1 ( 1)

8 [ 1] 1

n

jji i c t

A i key c n

* stands for the number of times the while loop test in line 5 is executed for that value of jt j

Example of analysis (average-case)

You should first figure out the distribution space of the instances of size n. Then compute an expected running time.Usually, we assume all possibilities are equally likely. Thus, in this example, in the while loop, on average, half of the elements in A[1, ..,j-1] are less than A[j], and half are greater than A[j]. Then on average, tj=j/2, and we can see the average-case time complexity is also quadratic.

Time complexity

Worst-case time complexity of an algorithm is the longest running time (or maximum number of elementary operations ) taken on any input of size n. E.g., the worst-case time complexity of INSERTION-SORT (or

the running time of INSERTION-SORT in the worst case) is (n2), which is an asymptotic tight bound.

The running time (time complexity) of INSERTION-SORT is O(n2), (n). But the running time (time complexity) of INSERTION-SORT is NOT (n2), NOT (n2), and is NOT O(n).

We cannot give a tight bound on the running time for all inputs. We usually consider one algorithm to be more efficient than

another if its worst-case running time has a lower order of growth.

• Since an algorithm's performance time may vary with different inputs of the same size, one commonly uses the worst-case time complexity of an algorithm, denoted as T(n), which is defined as the maximum amount of time taken on any input of size n.  

Worst-case time complexity

It is NOT true that each algorithm has a tight bound on its worst-case time complexity. E.g., suppose A is a sorted array of numbers, x is a number.n is the length of A. The following algorithm decides whether x is in A.

1. if n is odd k BINARYSEARCH (A, x) 2. else k LINEARSEARCH (A, x)

Obviously, the running time of this algorithm is O(n) for each input, and thus in the worst case.

And for each constant n0, there are infinitely many inputs whose size is larger than n0 and whose cost is lower-bounded by a constant times n, but, the running time in the worst case is NOT (n), and NOT (n), since, for each n0, there exists some nn0 such that no input of that size costs no less than a constant times n.

Input sizeInput size is the length of the string representing the input.Under TURING model, this is easy to define, that is the number of nonblank cells the input occupies on the input tape.In real world, this is impossible. The input size is not a precise measure of the input, and its interpretation is subject to the problem for which the algorithm is designed.Commonly used measures of input size are the following:

1.sorting and searching# of entries in the array or list 2.Graph algorithms # of vertices or edges, or both 3.Computational geometry # of points, vertices, edges, line

segments, polygons, etc. 4.Matrix operations dimensions of the input matrices 5.Number theory and cryptography # of bits in the input.

Input sizeInconsistencies brought about: An algorithm for adding two n n matrices which performs n2

additions sounds quadratic, while it is indeed linear in the input size.

The following two algorithms both compute the sum , and the # of elementary operations performed is the same, but they have different time complexity.

n

j

jA1

][

n

j

j1

Algorithm 2 input: A positive integer n Output: 1.sum0 2.for j1 to n 3. sum sum + j 4. end for 5.return sum

Algorithm 1 input: A positive integer n and an array A[1..n]with A[j]=j, 1j nOutput: 1.sum0 2.for j1 to n 3. sum sum + A[j] 4. end for 5.return sum

n

j

j1

Both algorithms run in time (n). But Algorithm 1 is a linear algorithmsince its time complexity is (n), where n is the real length of its input, whileAlgorithm 2 is an exponential algorithm since its input is a number whose lengthis k=log(n+1) , and (n)= (2k).