01 Analysis of Algorithms: Introduction

Analysis of AlgorithmsIntroduction

Andres Mendez-Vazquez

September 27, 2015

1 / 36

Outline1 Motivation

What is an Algorithm?Instance of a ProblemKolmogorov’s Definition

2 Problems Solved By AlgorithmsWhat Problems Are Solved By Algorithms?

3 SyllabusSubjects for the Class

4 Some Notes in NotationBefore Anything, The Notation for Pseudo-Code

5 What abstraction of a Computer to use?The Random-Access Machine (RAM) Machine

6 Analyzing AlgorithmsHow Input Size affects Running TimeThe First Method: Counting Number of OperationsCounting Equation For Insertion SortThe Analysis of the Best and Worst Case Inputs

2 / 36

Outline1 Motivation







3 / 36

Introduction

Informal definitionInformally, an algorithm is any well defined computational procedure that

It takes some value, or set of values, as input.Then, it produces some value, or set of values, as output.

Examples

4 / 36

Introduction



Examples

4 / 36

Introduction



Examples

4 / 36

Introduction



ExamplesALGORITHMINPUT

Cluster 1 (7)

OUTPUT

4 / 36

Example

Sorting ProblemInput: A sequence of N numbers a1, a2, ..., aN

Output: A reordering of the input sequence a(1), a(2), ..., a(N)

ActuallyWe are dealing with instances of a problem.

5 / 36

Example

Sorting ProblemInput: A sequence of N numbers a1, a2, ..., aN

Output: A reordering of the input sequence a(1), a(2), ..., a(N)

ActuallyWe are dealing with instances of a problem.

5 / 36

Outline1 Motivation







6 / 36

Instance of a Problem

Instance of the problemFor example, we have

I 9, 8, 5, 6, 7, 4, 3, 2, 1

Then, we finish withI 1, 2, 3, 4, 5, 6, 7, 8, 9

7 / 36

Although Instances are Important

NeverthelessThe way we use those instances is way more important

For exampleLook at Recursive Fibonacci!!!

8 / 36

Although Instances are Important

NeverthelessThe way we use those instances is way more important

For exampleLook at Recursive Fibonacci!!!

8 / 36

Example: Fibonacci

Fibonacci rule

Fn =

Fn−1 + Fn−2 if n > 11 if n = 10 if n = 0

Time Complexity1 Naive version using directly the recursion - exponential time.2 A more elegant version - linear time.

9 / 36

Example: Fibonacci

Fibonacci rule

Fn =

Fn−1 + Fn−2 if n > 11 if n = 10 if n = 0

Time Complexity1 Naive version using directly the recursion - exponential time.2 A more elegant version - linear time.

9 / 36

Outline1 Motivation







10 / 36

Kolmogorov’s DefinitionA bound for each sub-step

An algorithmic process splits into steps whose complexity is bounded in advanceI i.e., the bound is independent of the input and the current state of the

computation.

Transformations done at each stepEach step consists of a direct and immediate transformation of the current state.This transformation applies only to the active part of the state and does not alterthe remainder of the state.

Size of the stepsThe size of the active part is bounded in advance.

Ending the ProcessThe process runs until either the next step is impossible or a signal says thesolution has been reached.

11 / 36



computation.




11 / 36



computation.




11 / 36



computation.




11 / 36



computation.




11 / 36



computation.




11 / 36

BTW

How do they look this machines, this algorithms?After all we like to see them!!!

12 / 36

An Example of an Algorithm

Insertion Sort AlgorithmData: Unsorted Sequence AResult: Sort Sequence AInsertion Sort(A)for j ← 2 to lenght(A) do

key ← A[j];// Insert A[j] into the sorted sequence A[1, ..., j − 1]i ← j − 1;while i > 0 and A[i] > key do

A[i + 1]← A[i];i ← i − 1;

endA[i + 1]← key

end

13 / 36

Outline1 Motivation







14 / 36

Problems Solved By Algorithms

Single-SourceShortest Path.Solving systemsof linearequations.Huffman codesMatrixMultiplicationConvex hull

Short Paths in MapsThese algorithms allows to solve theproblem of finding the shortest pathin a map between two addresses.

@Copyright 2010-2011 Daniel Kastl,Frédéric Junod. Mis à jour le Apr 02, 2012.

15 / 36



Inverting MatricesNormally, given the system Ax = y,we do not use the inverse A−1, butthe LUP decomposition

L D U R

@Copyright From Wikimedia Commons, thefree media repository

15 / 36



Compression TreesThis method is part of the greedymethods. They are used forcompression, they can achieve 20%to 90% compression.

8

17

41

24

109

5

3 2 2 2 2

6

2 3 4

7

14

3

22

1 1

2

4

7

2

1111

4 4

' '

'U' 'A' 'L' 'M'

'E'

'D' 'F' 'N'

'S'

'B' 'H'

'O'

'J' 'P' 'R' 'T'


15 / 36



Fast Multiplication of MatricesIn many algorithms, we want tomultiply different n × n matrices

A11 A12 A21 A22

B11

B12

B21

B22

C11

11

C12 11

C21

11

C22 11

1

11

1

1 1

11

1 1

-1-1

1-1

1-1

1-1

1-1

1 1

-1-1

1

1

-1

-1

1

1

-1

-1

M1 M2 M3 M4 M5 M6 M7

STRASSEN'S ALGORITHM@Copyright From Wikimedia Commons, thefree media repository

15 / 36



Computational GeometryGiven the points in a plane, we wantto find the minimum convex hull thatencloses them.

p0

p1

p2

p3


15 / 36


Synthetic BiologyPattern RecognitionDatabasesFace Recognition

Computational Molecular EngineeringIn this field the engineers and biologist try to use thebasis of life to create complex molecular machines.All these machines will requiere complex algorithms

@Copyright 2002-2013 SYNTHETICCOMPONENTS NETWORK University of Bristol

16 / 36



Machine LearningIn pattern recognition, we try to find specificpatterns in data.

@2013Copyright Bülent Üstün, Department ofAnalytical Chemistry, University of Nijmegen

16 / 36



Partition of the Database SpaceA grid file or bucket grid is a point access method:

Query A1 = 'x' and A2 = 6

a f k p z

A2

A1

Grid Array

Pointers to buckets

0 2 7 10 15Intervals DefineSearch Areas

A1

Bucket Accessed

@Copyright Michael Unwalla: A mixed transactioncost model for coarse grained multi-columnpartitioning in a shared-nothing database machine

16 / 36



Person IdentificationFacial Recognition measure face landmarksto indentify different features in the face.

@Copyright 2013 Wouter Alberda, Olaf Kampinga

16 / 36

Outline1 Motivation







17 / 36

What will you learn in this class?1 Growth Functions2 Solving

Recursions3 Probabilistic

Analysis4 Sorting5 Median and

Order Statistics6 Review of Basic

Data Structures7 Advanced Data

Structures8 Advanced

Techniques

Basic ComplexitiesAsymptotic Notation - Ω, O andΘStandard notation and commonfunctions.

18 / 36







Techniques

For finding ComplexitiesThe substitution method.The recursive three method.The master method.

18 / 36







Techniques

Using Probability for findingComplexities

Indicator Random Variables.Randomization Algorithms.

18 / 36







Techniques

Sorting ThecniquesHeapsort.Quicksort.Sorting in linear time.

18 / 36







Techniques

Finding OrderMinimum and Maximum.Selection.Worst Case Selection.

18 / 36







Techniques

Basic Data StructuresElementary data structures.Hash tables.Applications of Hash Tables

I Linear CountingBinary search trees.

18 / 36







Techniques

Advanced Data StructuresB-Trees.Fibonacci Heaps.Data Structures for DisjointSets.

18 / 36







Techniques

Advanced TechniquesDynamic Programming.Greedy Algorithms.Amortized Analysis.Introduction to AnalyticCombinatronic

18 / 36

What will you learn in this class?

1 Graph Algorithms2 Selected Topics3 NP-Completeness4 Approximation

Algorithm

Graph AlgorithmsElementary Graph Algorithms.Minimum Spanning Trees.Strongly Connectedcomponents.Single-Source Shortest Paths.All-Pairs Shortest Paths.

19 / 36



Algorithm

ApplicationsMultithreaded Algorithms.String Matching.Computational Geometry

19 / 36



Algorithm

The Hard ProblemEncodings.Polynomial Time Verification.Polynomial Reduction.NP-Hard.NP-Complete proofs.A family of NP-Problems

19 / 36



Algorithm

Dealing with NPIf almost everything is NP, What todo?

BacktrackingBranch and BoundApproximation Algorithms

19 / 36

Now, what we are going to look at!!!

FirstSome stuff about notation!!!

SecondWhat abstraction of a Computer to use?

ThirdA first approach to analyzing algorithms!!!

20 / 36





20 / 36





20 / 36

Outline1 Motivation







21 / 36

Please follow these simple rules

Insertion Sort(A)1 for j ← 2 to length(A)2 do3 key ← A[j]4 IInsert A[j] into the sorted sequence A[1, ..., j − 1]5 i ← j − 16 while i > 0 and A[i] > key7 do8 A[i + 1]← A[i]9 i ← i − 110 A[i + 1]← key

RuleAlways put the name of the algorithm at the top.Together with the input.

22 / 36

Please follow these simple rulesInsertion Sort(A)

1 for j ← 2 to length(A)2 do3 key ← A[j]4 IInsert A[j] into the sorted sequence A[1, ..., j − 1]5 i ← j − 16 while i > 0 and A[i] > key7 do8 A[i + 1]← A[i]9 i ← i − 110 A[i + 1]← key

RuleAlways initialize all the variables.The a ← b means that the value b is passed to a. You also can use"=."

22 / 36



RuleUse identation to preserve the block structure avoiding clutter

22 / 36



RuleIt corresponds to comments. You can also use "//"

22 / 36

Outline1 Motivation







23 / 36

The Random-Access MachineDefinitionA Random Access Machine (RAM) is an abstract computational-machinemodel identical to a multiple-register counter machine with the addition ofindirect addressing.

InstructionsInstructions are executed one after another.

It contains arithmetic instructions found in low level languages.It has control instructions: Conditional and unconditional branches,return and call functions.It is able to do data movement: load, store, copy.It posses data types: integer and floating point.

Memory ModelA single block of memory is assumed.

24 / 36





24 / 36





24 / 36





24 / 36





24 / 36





24 / 36





24 / 36

RAM ModelWe have that

Control System

Memory With Random Access

Infinite

Register 1

Register 2

Register N

Inst

Inst

Inst

Inst

Inst

Inst

Inst

Inst

25 / 36

Although there are other equivalent models

Von Neumann architecture scheme

ArithmeticLogicUnit

ControlUnit

Memory

Input Output

Accumulator

26 / 36

Outline1 Motivation







27 / 36

Input Size and Running Time

DefinitionThe Input Size depends on the type of problem. We will indicate whichinput size is used per problem.

DefinitionThe Running Time of an algorithm is the number of of primitivesoperations or steps executed. For now, we will assume that each line in analgorithm takes ci a constant time.

28 / 36

Input Size and Running Time

DefinitionThe Input Size depends on the type of problem. We will indicate whichinput size is used per problem.

DefinitionThe Running Time of an algorithm is the number of of primitivesoperations or steps executed. For now, we will assume that each line in analgorithm takes ci a constant time.

28 / 36

Even Babbage cared about how many turns of the crankwere necessary!!!

Look at the crank!!!

29 / 36

Outline1 Motivation







30 / 36

Counting the Operations

Insertion Sort(A) I length(A):=N1 for j← 2 to length(A)2 do3 key ← A[j]4 I Insert A[j] into the sorted sequence A[1, ..., j − 1]5 i ← j − 16 while i > 0 and A[i] > key7 do8 A[i + 1]← A[i]9 i ← i − 110 A[i + 1]← key

Count Valuec1N

31 / 36



Count Valuec2(N − 1)

31 / 36




31 / 36



Count Valuec4∑N

j=2 j

31 / 36



Count Valuec5∑N

j=2(j − 1)

31 / 36



Count Valuec6∑N

j=2(j − 1)

31 / 36




31 / 36

Outline1 Motivation







32 / 36

Building a function for Analyzing Complexities

What can we do with all these quantities?Sum them all to get a step count that can be related to computationalcomplexities!!!

Counting EquationT (N ) = c1N + c2(N − 1) + c3(N − 1) + c4

(N(N+1)

2 − 1)

+ ...

c5(

N(N−1)2

)+ c6

(N(N−1)

2

)+ c7 (N − 1)

This can be reduced to something like...T (N ) = aN 2 + bN + c

33 / 36




(N(N+1)

2 − 1)

+ ...

c5(

N(N−1)2

)+ c6

(N(N−1)

2

)+ c7 (N − 1)


33 / 36




(N(N+1)

2 − 1)

+ ...

c5(

N(N−1)2

)+ c6

(N(N−1)

2

)+ c7 (N − 1)


33 / 36




(N(N+1)

2 − 1)

+ ...

c5(

N(N−1)2

)+ c6

(N(N−1)

2

)+ c7 (N − 1)


33 / 36

Outline1 Motivation







34 / 36

The Worst and The Average

1 The Worst Case2 The Average

Case

Worst SequenceUpper bound on the runningtime of an algorithm.In case of insertion sort, it willbe the permutation:N , N − 1, N − 2, ..., 3, 2, 1

35 / 36

The Worst and The Average

1 The Worst Case2 The Average

Case

Average SequenceIn the case of insertion sort,when half of the elements ofA[1, 2, ..., j − 1] are less thanA[j] and half are greater.Then, there is only the need tocheck half of the elements i.e.j2 .

35 / 36

WHY TO LOOK FOR EFFICIENT ALGORITHMS?

Example1 Insertion Sort2 Merge Sort3 Constraints

for theexample

4 Final Result

ExampleT (N ) = c1N 2

36 / 36



for theexample

4 Final Result

ExampleT (N ) = c2N log2 N

36 / 36



for theexample

4 Final Result

ExampleAssume, we have 106 numbers tosort, c1 = 2 for a supercomputerand c2 = 50 for a PC.The supercomputer can do 1010

instructions per second.The PC can do 107 instructions persecond.

36 / 36



for theexample

4 Final Result

ExampleInsertion Sort in the Supercomputer

2(106)2 ins1010 ins/sec = 200sec

Merge Sort in the Humble PC

2(106) log(106) ins107 ins/sec = 3.9sec

36 / 36