Analyzing Algorithms Growth of Functions

CSE 5350 - Fall 2007Analyzing AlgorithmsGrowth of Functions Slide 1

Analyzing AlgorithmsAnalyzing AlgorithmsGrowth of FunctionsGrowth of FunctionsAnalyzing AlgorithmsAnalyzing AlgorithmsGrowth of FunctionsGrowth of Functions

(Chapters 1 & 2)

Mihaela IridonMihaela Iridon, Ph.D.

[email protected]

CSE 5350/7350Introduction to Algorithms


What means analyzing What means analyzing algorithms?algorithms?• Predicting the required resources• What do we measure?

– Computational time– Memory– Communication bandwidth– Other

• Model constructs:– Technology– Resources (hardware & software)– Associated costs– Assumptions (1 processor, RAM-model: sequential

operations)


Tools used for algorithm Tools used for algorithm analysisanalysis• Mathematical tools:

– Discrete combinatorics– Probability theory– Ability to single out the predominant

operations (most significant terms in a formula)

• Modeling and simulation tools• Software utilities, benchmarking

programs


TerminologyTerminology

• Input Size– In general the time to execute a set of

operations is dependent on the size of the input

– Depends on the problem definition– = the number of items in the input– Could be more than one number (e.g. a graph)

• Running Time– = the number of primitive operations (steps)

executed– Machine-independent term


Example (Example (11) – Insertion Sort () – Insertion Sort (In In PlacePlace))

/// <summary> /// Sorts the input list of integers by using the Insertion Sorting algorithm /// (see Cormen textbook, Chapter 1.1) /// </summary> /// <param name="input">Input list of integers (to be sorted) – input list will be modified</param> public static void InsertionSort(List<int> input) { if (input == null) return; //NULL input if (input.Count < 2) return; //one-element array; nothing to sort

int i=0, key=0; for (int j = 1; j < input.Count; j++) { key = input[j];

//insert input[j] into the sorted sequence input[0..j-1] i = j-1; while (i > -1 && input[i] > key) { input[i+1] = input[i]; i--; } input[i+1] = key; } }

/// <summary> /// Sorts the input list of integers by using the Insertion Sorting algorithm /// (see Cormen textbook, Chapter 1.1) /// </summary> /// <param name="input">Input list of integers (to be sorted) – input list will be modified</param> public static void InsertionSort(List<int> input) { if (input == null) return; //NULL input if (input.Count < 2) return; //one-element array; nothing to sort

int i=0, key=0; for (int j = 1; j < input.Count; j++) { key = input[j];

//insert input[j] into the sorted sequence input[0..j-1] i = j-1; while (i > -1 && input[i] > key) { input[i+1] = input[i]; i--; } input[i+1] = key; } }

Cost # of Times executedC1 n

C2 n-1

0 n-1C4 n-1C5 Σ(j=1..n-1)tj

C6 Σ(j=1..n-1)(tj – 1)C7 Σ(j=1..n-1)(tj – 1)

C8 n-1

Cost # of Times executedC1 n

C2 n-1

0 n-1C4 n-1C5 Σ(j=1..n-1)tj

C6 Σ(j=1..n-1)(tj – 1)C7 Σ(j=1..n-1)(tj – 1)

C8 n-1

1

1

n

jjt


Example (Example (11) – Insertion Sort () – Insertion Sort (In In PlacePlace))• The running time depends on the input

value– If the input is already sorted then the body of the while loop does

not execute and the best case scenario/running time for insertion sort is:

T(n) = c1 n + c2 (n-1) + c4 (n-1) + c5 (n-1) + c8 (n-1) = (c1 + c2 + c4 + c5 + c8) * n – (c2 + c4 + c5 +

c8) = a * n + b linear function of n

– If the input is in reverse sorted order: (worst case scenario)

T(n) = c1 n + c2 (n-1) + c4 (n-1) + c5 (n(n+1)/2 - 1) +

c6 [n(n-1)/2] + c7 [n(n-1)/2] + c8 (n-1) = a * n2 + b * n + c quadratic function

of n


Worst-case & Average-case Worst-case & Average-case AnalysisAnalysis

• Worst-case running time:– The longest running time for any input of size n

(i.e. the longest path in the execution)

– Upper bound on the running time for any input– Occurs fairly often– The average-case ~ the worst-case

• Average-case running time:– Difficult to define what average input means– Example for Insertion Sort: On average, half the

elements in an array A1 ... Aj-1 are less than an element Aj, and half are greater.


Order/Rate of GrowthOrder/Rate of Growth

• Simplifying abstraction• Consider only the leading term• Ignore the leading term’s constant

coefficient• Worst-case running time for

Insertion Sort is Θ(n2) (theta of n-squared)

• An algorithm with Θ(n2) will run faster than one with Θ(n3)


Divide-and-Conquer AlgorithmsDivide-and-Conquer Algorithms

• Incremental approach (e.g. Insertion Sort)

• Divide-and-conquer approach (e.g. recursive algorithms such as Merge Sort)– [Divide] Break the problem into related

or similar sub-problems of smaller size;– [Conquer] Solve the sub-problems– [Combine] Combine the solutions


Analyzing divide-and-conquer Analyzing divide-and-conquer algorithmsalgorithms• Recursive approach use recurrence

equation (recurrence) to describe the running time

• T(n) = running time on a problem of size n• If n= small (n <= c) then Θ(1)• Otherwise, divide problem in a sub-

problems, each of size n / b• D(n) = time to divide• C(n) = time to combine• T(n) = a*T(n/b) + D(n) + C(n) (when n > c)


Merge Sort (1)Merge Sort (1)

public static void MergeSort2(List<int> input, int startIx, int endIx)

{

if (input == null) return;

if (startIx == endIx) return; //stop condition

int middle = (int) Math.Floor((endIx + startIx) / 2.0);

MergeSort2(input, startIx, middle);

MergeSort2(input, middle + 1, endIx);

Combine2(input, startIx, middle, endIx);

}

public static void MergeSort2(List<int> input, int startIx, int endIx)

{

if (input == null) return;

if (startIx == endIx) return; //stop condition

int middle = (int) Math.Floor((endIx + startIx) / 2.0);

MergeSort2(input, startIx, middle);

MergeSort2(input, middle + 1, endIx);

Combine2(input, startIx, middle, endIx);

}


Merge Sort (2)Merge Sort (2)public static void Combine2(List<int> input, int i1, int i2, int i3)

{

if (input == null || input.Count == 0) return;

List<int> result = new List<int>(i3 - i1 + 1); //not 100% in-place

int ix1 = i1, ix2 = i2+1;

while (result.Count < i3 - i1 + 1)

{

while (ix1 < i2 + 1 && (ix2 == (i3 + 1) || input[ix1] < input[ix2]))

result.Add(input[ix1++]);

while (ix2 < i3 + 1 && (ix1 == (i2 + 1) || input[ix1] > input[ix2]))


}

for (int j = i1; j <= i3; j++)

input[j] = result[j - i1];

}

public static void Combine2(List<int> input, int i1, int i2, int i3)

{

if (input == null || input.Count == 0) return;

List<int> result = new List<int>(i3 - i1 + 1); //not 100% in-place

int ix1 = i1, ix2 = i2+1;

while (result.Count < i3 - i1 + 1)

{

while (ix1 < i2 + 1 && (ix2 == (i3 + 1) || input[ix1] < input[ix2]))


while (ix2 < i3 + 1 && (ix1 == (i2 + 1) || input[ix1] > input[ix2]))


}

for (int j = i1; j <= i3; j++)

input[j] = result[j - i1];

}


Analyzing Merge SortAnalyzing Merge Sort

• [Divide] D(n) = Θ(1)• [Conquer] 2 * T(n/2)• [Combine] C(n) = Θ(n)

• T(n) = Θ(1) if n=12T(n/2) + Θ(n) if n≥1= Θ(n * log2 n) better than Insertion

Sort

{


Growth of FunctionsGrowth of Functions

• Algorithms efficiency• Compare relative performance of

alternative algorithms• Analysis for large input size: e.g. n ∞• Asymptotic efficiency of algorithms:

– Input size is large enough to make only the order of growth of the running time relevant

– How the running time increases with the size of the input in the limit, as the size of the input increases without bound


Asymptotic NotationAsymptotic Notation

• Notations used to describe the asymptotic running time of an algorithm

• Are defined in terms of functions whose domains are the set N = {0, 1, 2, …}

• Convenient for describing the worst-case running-time function T(n)

• Notation abused vs. misused


ΘΘ-notation-notation• Θ: asymptotically bounds a function from

above and below

f(n) = Θ(g(n)) indicates f(n) Θ(g(n))

org(n) is an asymptotically tight bound for f(n)

})()()(0

0,,:)({))((

021

021

nnngcnfngc

nccnfng


OO-notation-notation• O: asymptotic upper bound

f(n) = Θ(g(n)) f(n) = O(g(n))• The Θ-notation is stronger than the O-notation• Example: n = O(n2)• E.g.: worst-case for insertion sort = O(n2)• Notation abuse: The running time of insertion sort is

O(n2). – The running time depends on the particular input of size n. – It is true only for the worst-case scenario (i.e. no matter what

particular input of size n is chosen for each value of n)

})()(0

0,:)({))((

0

0

nnncgnf

ncnfngO


ΩΩ-notation-notation• Ω: asymptotic lower bound

• Theorem f(n) and g(n), f(n) = Θ(g(n)) iff

f(n) = O(g(n)) and f(n) = Ω(g(n))

• Used to bound the best-case running time• E.g.: best-case for insertion sort is Ω(n)• Worst-case for insertion sort is Ω(n2)

})()(0

0,:)({))((

0

0

nnnfncg

ncnfng


Graphical ComparisonGraphical Comparison


oo-notation-notation

• Upper bound that is not asymptotically tight

• 2n2 = O(n2) is asymptotically tight• 2n = O(n2) is not asymptotically tight

• 2n = o(n2), but 2n2 o(n2)

})()(0

0,0:)({))((

0

0

nnncgnf

ncnfngo

0)(

)(lim

ng

nfn


-notation-notation

• Lower bound that is not asymptotically tight

• f(n) (g(n)) iff g(n) o(f(n))

• E.g.: n2/2 = (n), but n2/2 = (n2)

})()(0

0,0:)({))((

0

0

nnnfncg

ncnfng

)(

)(lim

ng

nfn


Comparison of FunctionsComparison of Functions

• Transitivity (for all five notations):If f(n) = X(g(n)) and g(n) = X(h(n)) f(n)

= X(h(n))• Reflexivity (for the big-X notations)

f(n) = X(f(n))• Symmetry:

f(n) = Θ(g(n)) iff g(n) = Θ(f(n))• Transpose symmetry:

f(n) = O(g(n)) iff g(n) = Ω(n)f(n) = o(g(n)) iff g(n) = (n)


Analyzing Algorithms - Analyzing Algorithms - AddendumAddendum

•Finding the largest clique in a graph•Parsing an object model


Finding the largest cliqueFinding the largest clique• Graph: G(V,E)• A graph or undirected graph G is an

ordered pair G: = (V,E) that is subject to the following conditions:

– V is a set, whose elements are called vertices or nodes, – E is a set of pairs (unordered) of distinct vertices, called

edges or lines.

• A clique in an undirected graph G is a set of vertices V such that for every two vertices in V, there exists an edge connecting the two. (Complete sub-graph)


The Clique ProblemThe Clique Problem

• determining whether a graph contains a clique of at least a given size k.

• Verification of actual clique : trivial• The clique problem is in NP (non-

deterministic polynomial time).• NP-complete • The corresponding optimization

problem, the maximum clique problem, is to find the largest clique in a graph.


Brute force approach to the clique Brute force approach to the clique problemproblem• Examine each sub-graph with at least k

vertices and check to see if it forms a clique• Number of cases to inspect:

• A clique C=(vi1, vi2, .., vin) exists only when its n sub-cliques each of size n-1 exist.

• Event-raising mechanism to increment a counter of sub-cliques using the threshold graph

• Space required to build all cliques until G=completely connected: 12..

432

n

n

nnnn n


Sorting an object modelSorting an object model

• Input:– A collection of property paths

• Output:– A sorted collection of property paths

Input Sample (excerpt)Input Sample (excerpt)• Company.Principals[0].References[0].ResidentialInfos[0].Address.AddressType• Company.Principals[0].References[0].ResidentialInfos[1].Address.AddressType• Company.Principals[0].References[1].ResidentialInfos[0].Address.AddressType• Company.Principals[0].References[1].ResidentialInfos[1].Address.AddressType• Company.Principals[1].References[0].ResidentialInfos[0].Address.AddressType• Company.Principals[1].References[0].ResidentialInfos[1].Address.AddressType• Company.Principals[1].References[1].ResidentialInfos[0].Address.AddressType• Company.Principals[1].References[1].ResidentialInfos[1].Address.AddressType• Company.Principals[0].References[0].ResidentialInfos[0].Address.StreetAddressLine1• Company.Principals[0].References[0].ResidentialInfos[1].Address.StreetAddressLine1• Company.Principals[0].References[1].ResidentialInfos[0].Address.StreetAddressLine1• Company.Principals[0].References[1].ResidentialInfos[1].Address.StreetAddressLine1• Company.Principals[1].References[0].ResidentialInfos[0].Address.StreetAddressLine1• Company.Principals[1].References[0].ResidentialInfos[1].Address.StreetAddressLine1• Company.Principals[1].References[1].ResidentialInfos[0].Address.StreetAddressLine1• Company.Principals[1].References[1].ResidentialInfos[1].Address.StreetAddressLine1• Company.Principals[0].References[0].ResidentialInfos[0].Address.StreetAddressLine2• Company.Principals[0].References[0].ResidentialInfos[1].Address.StreetAddressLine2• Company.Principals[0].References[1].ResidentialInfos[0].Address.StreetAddressLine2• Company.Principals[0].References[1].ResidentialInfos[1].Address.StreetAddressLine2• Company.Principals[1].References[0].ResidentialInfos[0].Address.StreetAddressLine2• Company.Principals[1].References[0].ResidentialInfos[1].Address.StreetAddressLine2• Company.Principals[1].References[1].ResidentialInfos[0].Address.StreetAddressLine2• Company.Principals[1].References[1].ResidentialInfos[1].Address.StreetAddressLine2


Object ModelObject Model


CompanyPrincipals[]FinancialStatements[]TradeLines[]Contacts[]Declarations[]Documents[]Addresses[]CompanyInfoBusinessAddressMailingAddressRelationshipSummary

PrincipalEmploymentInfos[]IncomeInfos[]TradeLines[]References[]Assets[]IdentificationInfos[]ResidentialInfos[]ContactInfoPersonalInfoPrincipalTypeEmployedByCompanyYearsAsOwnerIndividualOrJointTypePersonType

AddressAddressTypeStreetAddressLine1StreetAddressLine2CityCountyStateZipcodeCountry

1 1..*

1..*

1

Output sample (sorted object Output sample (sorted object model)model)• Company.Principals[0].EmploymentInfos[0].EmployerAddress.AddressType

• Company.Principals[0].EmploymentInfos[0].EmployerAddress.StreetAddressLine1• Company.Principals[0].EmploymentInfos[0].EmployerAddress.StreetAddressLine2• Company.Principals[0].EmploymentInfos[0].EmployerAddress.City• Company.Principals[0].EmploymentInfos[0].EmployerAddress.County• Company.Principals[0].EmploymentInfos[0].EmployerAddress.State• Company.Principals[0].EmploymentInfos[0].EmployerAddress.ZipCode• Company.Principals[0].EmploymentInfos[0].EmployerAddress.Country• Company.Principals[0].EmploymentInfos[0].EmployerName• Company.Principals[0].EmploymentInfos[0].EmploymentType• Company.Principals[0].EmploymentInfos[0].OccupationType• Company.Principals[0].EmploymentInfos[0].YearsOfEmployment• Company.Principals[0].EmploymentInfos[0].MonthsOfEmployment• Company.Principals[0].EmploymentInfos[0].Title• Company.Principals[0].EmploymentInfos[0].Department• Company.Principals[0].EmploymentInfos[1].EmployerAddress.AddressType• Company.Principals[0].EmploymentInfos[1].EmployerAddress.StreetAddressLine1• Company.Principals[0].EmploymentInfos[1].EmployerAddress.StreetAddressLine2• Company.Principals[0].EmploymentInfos[1].EmployerAddress.City• Company.Principals[0].EmploymentInfos[1].EmployerAddress.County• Company.Principals[0].EmploymentInfos[1].EmployerAddress.State• Company.Principals[0].EmploymentInfos[1].EmployerAddress.ZipCode• Company.Principals[0].EmploymentInfos[1].EmployerAddress.Country• Company.Principals[0].EmploymentInfos[1].EmployerName


Data GraphData Graph


ROOT ApplicationNumber

ConversationLogs Comment

DateTimeStamp

IncludeInUDR

Company Principals EmploymentInfos

FinancialStatements

IncomeInfos

TradeLines

Data StructuresData Structures


Node

Text : stringCount : intCrtIndex : intIsLast : boolNextNode : NodeRightNode : Node