Algorithms and Data Structures - vowi.fsinf.at¼hrung_in_C(plusplus... · Algorithms and Data Structures Markus Würzl, Vienna University of Technology Solving Interesting Problems

Algorithms and DataStructures

Markus Würzl, Vienna University of Technology

Solving Interesting Problems1. Create data structures & algorithms to solve

problems.

2. Prove algorithms work.• Buggy algorithms are worthless!

3. Examine properties of algorithms.• Simplicity, running time, space needed, …

A problem instance is a specific task to be solved on a specific input.

A problem is a collection of similar problem instances.An algorithm is correct for a given problem instance

if it produces the desired output for this instance.An algorithm is correct for a problem P (solves

problem P) if it is correct for all instances of P.

When Does an Algorithm Solve a Problem?

What Kinds of Problems?Decision problems

Given input, does it have a certain property?

– Is an integer prime?– Is a graph planar? (Can be drawn on a plane

without overlapping edges?)

What Kinds of Problems?

Function problems

Given input, compute the unique solution.

– Compute the product of two matrices.– Sort an array.

What Kinds of Problems?Search problems

Given input, compute any one of a set of solutions.

– Find the factors of an integer.– Find shortest path between two points in graph.– Find shortest path between each pair of points in graph.– Find longest path between two points in graph.

Properties of Algorithms

Time efficiency– As a function of its input size, how long does it

take?

Space efficiency– As a function of its input size, how much additional

space does it use?

Computational resources = CPU time (running time), memory usage (space), messages sent along the network, …

Resource consumption differs depending on the size of the input (length of text, number of records to be sorted, …)

Specify resource consumption as a function of the inputs size.

Resource consumption may even differ greatly for inputs of the same size, depending on their structure (highly unsorted input, almost sorted input, …)

Analysis of Computational Resources

Properties of Algorithms

Simplicity– Informal notion– Pragmatic benefits

How easy to code correctly?Possibly low constant factors in efficiency

– Idealistic benefitMathematical “elegance”

Best, Average, and Worst Case

The best-case resource consumption is the minimumresource consumption over all possible inputs of a given size.

The worst-case resource consumption is the maximum resource consumption over all inputs of a given size.

The average-case resource consumption is the average resource consumption over all possible inputs of a given size.

Algorithm: Description of a procedure to solve a given problem

Model of computation: Specifies permissible operations and their costs

An algorithm is correct = produces correct answer for every possible instance of the problem

Resource consumption = function of input (size)Best-case, worst-case, average-case analysis

Things to Remember

Extended Example: Multiplication

Illustrates some of course’s interesting ideas.Just an overview – not too much detail yet.

How to multiply two integers: x×y.Common & familiar. Simple?

Suggested algorithms & efficiency?

? ??

Multiplication Algorithm 1

It’s a single basic machine operation, O(1) time.

Realistic only if we have bounded-sized integers.

Instead, assume unbounded-sized integers.E.g., as in Scheme.Explicitly include this assumption.

Problem with this answer? ??

Multiplication Algorithm 2Repeated addition: x × y = x+…+x = y+…+y

y copies x copies

Back to multiplication:• O(n × max(x,y)) = O(n × 2n)

How long for addition?• Grade-school algorithm takes time ∝ number length.• Define: n = log2(max(x,y))

• Logarithm base irrelevant, since O(logc x) = O(logc’ x)

• O(n)

Multiplication Algorithm 3

Grade-school “long-hand” algorithm.

n multiplications of (x by 1 bit of y) +n additions of the resulting products.

Total?

x= 38

y= × 473114

2660+ 15200

17974

Much better. Also approximates hardware algorithms.O(n×n + n×n) = O(n2)

??

Multiplication Algorithm 4: Part 1

Karatsuba’s algorithm, 1962

Break the bitstrings of x,y into halves.x = a × 2n/2 + b = a << n/2 + by = c × 2n/2 + d = c << n/2 + d

1 0 1 1 0 1

1 1 1 0 0

x=45

y=28

a=5

c=3

b=5

d=4xy = ac << n + (ad+bc) << n/2 + bd

Compute the 4 subproducts recursively.

Example of a "divide-and-conquer" algorithm.


How long does this take?k = arbitrary

constant to fill in for the details

4 recursive subproducts +

additions & shifts

Solve recurrence equation.• How? Discussed next week.• T(n) = O(n2)

No better than previous algorithm.

( )⎪⎩

⎪⎨⎧

>×+⎟⎠⎞

⎜⎝⎛×

== 1nnk

2n

T4

1nknT

Form recurrence equation:

Multiplication Algorithm 4: Part 2Previous:

xy = ac << n + (ad+bc) << n/2 + bd

Regroup:u = (a+b) × (c+d)v = acw = bdxy = v << n + (u-v-w) << n/2 + w

Only 3 subproducts!

1 0 1 1 0 1

1 1 1 0 0

x=45

y=28

a=5

c=3

b=5

d=4


How long does this take?( )

⎪⎩

⎪⎨⎧

>×′+⎟⎠⎞

⎜⎝⎛′×

=′=′ 1nnk

2n

T3

1nknT

k’ = a new, larger constant

( ) ( ) ( )1.593log3log nOnOnk2nk3nT 22 ≈=×′×−×′×=′

More complicated, but asymptotically faster.

Multiplication Algorithm 4: Part 2Previous:

u = (a+b) × (c+d) v = ac w = bdxy = v << n + (u-v-w) << n/2 + w

An overlooked detail:a+b and c+d can be n/2+1 bit numbers.Middle term can overflow into high term.

Break sums into (n/2+1)th

bit and rest.

Solution:a+b = a1 << n/2 + b1

c+d = c1 << n/2 + d1

(a+b) × (c+d) = (a1 c1) << n + (a1 d1 + b1 c1) << n/2 + b1 d1

O(1) time O(n/2) time

Multiplication Algorithms 5 & 6

Karp’s FFT-based algorithm: O(n log2 n)

Schoenhage & Strassen’s FFT-based algorithm, 1971: O(n log n log log n)– Example of log-shaving – slightly better running-

times with lower logarithmic terms.– Best known.

Multiplication Algorithms 7, ...

Use parallelism.

Even serial processors use some bit-level parallelism.Divide-&-conquer & FFT algorithms easily parallelized.

Beyond the scope of this course.

Multiplication Algorithms Summary

Higher constant factors →more complicated, asymptotically-faster algorithms are

slower for typical input sizes.

Karatsuba’s algorithm somewhat practical:– Used on large numbers in cryptography.– Generalizes to matrix multiplication

Strassen’s algorithm, [CLRS] 28.2

Algorithm Analysis Summary1. State problem.2. Characterize the input size.3. State algorithm.

– Often difficult, of course.4. Prove algorithm correctness.

– Necessary!5. Determine its resource usage.

– Often via recurrence equations.– Allows comparison of algorithms.– Can decide which algorithm suitable for given application.– Can guide the search for better algorithms.

We almost missed an important detail that would have produced

incorrect result.

Lists

Abstract Data Structures

Abstractly, many data structures are simply collections of (key, value) pairs.

Typically, interested in some of the following operations…

Basic Operations

Almost always include these.

Create : void → collection

Insert : collection × kv → void

Search : collection × key → kv

…

A (key,value) pair or a pointer to a pair.

Alternatively, just a value.

Additional Operations

Often include some of these.

Min : collection → kvMax : collection → kvPred : collection × kv → kvSucc : collection × kv → kv

Union : collection × collection → voidDelete : collection × kv → void

Requires keys from a totally ordered set.

Dictionaries

A particularly common set of operations:

CreateInsertDeleteSearch

Concrete Data Structures

Will be looking at concrete data structures & their costs to implement those operations.

First, variations of lists:– Singly-linked– Singly-linked & sorted– Doubly-linked– Doubly-linked & sorted

3,x 6,a 1,b 2,k

3,x 6,a 1,b 2,k

3,x 6,a1,b 2,k

3,x 6,a1,b 2,k

Mutability Reminder

Unlike earlier courses, this course emphasizes mutable data structures.

Why?– Largely tradition.– Immutable data structures sometimes less

asymptotically efficient.– Immutable data structures much less studied.

I recommend “Purely Functional Data Structures”, Okasaki 1998.

Worst-Case Time Complexities

Search

Insert

Create

Doubly, sorted

DoublySingly, sorted

SinglyLet n = collection size.

1 1 1 1 Creation almost always O(1).

1 1n n Linear time to maintain sortedness.

n n nn Linear search is only option.

Succ

Pred

Max

Min

Doubly, sorted


Singly

Sortedness helps.n 1 1n

Must search, unless given ptr to kv as arg.n 1 or n 1 or nn

Must search, unless given ptrto kv as arg & previous ptr

points to pred.n n 1 or nn

Tail ptr helps if sorted.

n 1 or n1 or n

n

Delete

Union

Doubly, sorted


Singly

1 or n n n1 or n

n n 1 or n1 or n Must search, unless given ptr to kv as arg & have

previous ptrs in list.

Unsorted → append. Sorted → merge.

Tail ptr helps append.

LessonsThere are many kinds of “lists”.

To get lower complexities, generally need– more space (double-linkage and/or tail pointers) and– larger constant factors (maintaining double-linkage).

Extra invariants sometimes help, sometimes hurt.

Which is best depends on the operations needed.

I.e., there are always tradeoffs.

Other Concrete Data Structures

? What about arrays? ?Some advantages, but imposes maximum collection size.

But, can occasionally enlargen array efficiently.Requires “amortized” analysis – discussed later.

Can combine arrays + lists into hash tables.Discussed later.


? What about trees? ?

Again, can be singly- or doubly-linked, unsorted or sorted.

Worst-case bounds same as lists.Balancing helps – discussed soon…


? What about heaps? ?

Some operations logarithmic:Insert, either Min or Max, either DeleteMin or DeleteMax.

But Union is linear.

Will discuss extensions to heaps to improve on this.

1

Elementary Data Structures

Stacks, Queues, & ListsAmortized analysisTrees

Elementary Data Structures 2

The Stack ADT (§4.2.1)The Stack ADT stores arbitrary objectsInsertions and deletions follow the last-in first-out schemeThink of a spring-loaded plate dispenserMain stack operations:

push(Object o): inserts element opop(): removes and returns the last inserted element

Auxiliary stack operations:

top(): returns the last inserted element without removing itsize(): returns the number of elements storedisEmpty(): a Boolean value indicating whether no elements are stored


Applications of Stacks

Direct applicationsPage-visited history in a Web browserUndo sequence in a text editorChain of method calls in the Java Virtual Machine or C++ runtime environment

Indirect applicationsAuxiliary data structure for algorithmsComponent of other data structures


Array-based Stack (§4.2.2)

A simple way of implementing the Stack ADT uses an arrayWe add elements from left to rightA variable t keeps track of the index of the top element (size is t+1)

S0 1 2 t

…

Algorithm pop():if isEmpty() then

throw EmptyStackExceptionelse t ← t − 1return S[t + 1]

Algorithm push(o)if t = S.length − 1 then

throw FullStackExceptionelse t ← t + 1S[t] ← o


Growable Array-based Stack

In a push operation, when the array is full, instead of throwing an exception, we can replace the array with a larger oneHow large should the new array be?

incremental strategy: increase the size by a constant cdoubling strategy: double the size

Algorithm push(o)if t = S.length − 1 then

A ← new array ofsize …

for i ← 0 to t doA[i] ← S[i]S ← A

t ← t + 1S[t] ← o


Comparison of the Strategies

We compare the incremental strategy and the doubling strategy by analyzing the total time T(n) needed to perform a series of npush operationsWe assume that we start with an empty stack represented by an array of size 1We call amortized time of a push operation the average time taken by a push over the series of operations, i.e., T(n)/n

2


Analysis of the Incremental Strategy

We replace the array k = n/c timesThe total time T(n) of a series of n push operations is proportional to

n + c + 2c + 3c + 4c + … + kc =n + c(1 + 2 + 3 + … + k) =

n + ck(k + 1)/2Since c is a constant, T(n) is O(n + k2), i.e., O(n2)The amortized time of a push operation is O(n)


Direct Analysis of the Doubling Strategy

We replace the array k = log2 n timesThe total time T(n) of a series of n push operations is proportional to

n + 1 + 2 + 4 + 8 + …+ 2k =n + 2k + 1 −1 = 2n −1

T(n) is O(n)The amortized time of a push operation is O(1)

geometric series

1

2

14

8


The accounting method determines the amortized running time with a system of credits and debitsWe view a computer as a coin-operated device requiring 1 cyber-dollar for a constant amount of computing.

Accounting Method Analysis of the Doubling Strategy

We set up a scheme for charging operations. This is known as an amortization scheme.The scheme must give us always enough money to pay for the actual cost of the operation.The total cost of the series of operations is no more than the total amount charged.

(amortized time) ≤ (total $ charged) / (# operations)


Amortization Scheme for the Doubling Strategy

Consider again the k phases, where each phase consisting of twice as many pushes as the one before.At the end of a phase we must have saved enough to pay for the array-growing push of the next phase.At the end of phase i we want to have saved i cyber-dollars, to pay for the array growth for the beginning of the next phase.

0 2 4 5 6 731

$ $ $ $$ $ $ $

0 2 4 5 6 7 8 9 113 10 12 13 14 151

$$

• We charge $3 for a push. The $2 saved for a regular push are “stored” in the second half of the array. Thus, we will have 2(i/2)=i cyber-dollars saved at then end of phase i.• Therefore, each push runs in O(1) amortized time; n pushes run in O(n) time.


The Queue ADT (§4.3.1)The Queue ADT stores arbitrary objectsInsertions and deletions follow the first-in first-out schemeInsertions are at the rear of the queue and removals are at the front of the queueMain queue operations:

enqueue(object o): inserts element o at the end of the queuedequeue(): removes and returns the element at the front of the queue

Auxiliary queue operations:front(): returns the element at the front without removing itsize(): returns the number of elements storedisEmpty(): returns a Boolean value indicating whether no elements are stored

ExceptionsAttempting the execution of dequeue or front on an empty queue throws an EmptyQueueException


Applications of Queues

Direct applicationsWaiting linesAccess to shared resources (e.g., printer)Multiprogramming

Indirect applicationsAuxiliary data structure for algorithmsComponent of other data structures

3


Singly Linked ListA singly linked list is a concrete data structure consisting of a sequence of nodesEach node stores

elementlink to the next node

next

elem node

A B C D

∅


Queue with a Singly Linked ListWe can implement a queue with a singly linked list

The front element is stored at the first nodeThe rear element is stored at the last node

The space used is O(n) and each operation of the Queue ADT takes O(1) time

f

r

∅

nodes

elements


List ADT (§5.2.2)

The List ADT models a sequence of positionsstoring arbitrary objectsIt allows for insertion and removal in the “middle” Query methods:

isFirst(p), isLast(p)

Accessor methods:first(), last()before(p), after(p)

Update methods:replaceElement(p, o), swapElements(p, q) insertBefore(p, o), insertAfter(p, o),insertFirst(o), insertLast(o)remove(p)


Doubly Linked ListA doubly linked list provides a natural implementation of the List ADTNodes implement Position and store:

elementlink to the previous nodelink to the next node

Special trailer and header nodes

prev next

elem

trailerheader nodes/positions

elements

node


Trees (§6.1)In computer science, a tree is an abstract model of a hierarchical structureA tree consists of nodes with a parent-child relationApplications:

Organization chartsFile systemsProgramming environments

Computers”R”Us

Sales R&DManufacturing

Laptops DesktopsUS International

Europe Asia Canada


Tree ADT (§6.1.2)We use positions to abstract nodesGeneric methods:

integer size()boolean isEmpty()objectIterator elements()positionIterator positions()

Accessor methods:position root()position parent(p)positionIterator children(p)

Query methods:boolean isInternal(p)boolean isExternal(p)boolean isRoot(p)

Update methods:swapElements(p, q)object replaceElement(p, o)

Additional update methods may be defined by data structures implementing the Tree ADT

4


Preorder Traversal (§6.2.3)A traversal visits the nodes of a tree in a systematic mannerIn a preorder traversal, a node is visited before its descendants Application: print a structured document

Make Money Fast!

1. Motivations References2. Methods

2.1 StockFraud

2.2 PonziScheme1.1 Greed 1.2 Avidity 2.3 Bank

Robbery

1

2

3

5

4 6 7 8

9

Algorithm preOrder(v)visit(v)for each child w of v

preorder (w)


Postorder Traversal (§6.2.4)In a postorder traversal, a node is visited after its descendantsApplication: compute space used by files in a directory and its subdirectories

Algorithm postOrder(v)for each child w of v

postOrder (w)visit(v)

cs16/

homeworks/ todo.txt1Kprograms/

DDR.java10K

Stocks.java25K

h1c.doc3K

h1nc.doc2K

Robot.java20K

9

3

1

7

2 4 5 6

8


Amortized Analysis of Tree Traversal

Time taken in preorder or postorder traversal of an n-node tree is proportional to the sum, taken over each node v in the tree, of the time needed for the recursive call for v.

The call for v costs $(cv + 1), where cv is the number of children of vFor the call for v, charge one cyber-dollar to v and charge one cyber-dollar to each child of v.Each node (except the root) gets charged twice: once for its own call and once for its parent’s call.Therefore, traversal time is O(n).


Binary Trees (§6.3)A binary tree is a tree with the following properties:

Each internal node has two childrenThe children of a node are an ordered pair

We call the children of an internal node left child and right childAlternative recursive definition: a binary tree is either

a tree consisting of a single node, ora tree whose root has an ordered pair of children, each of which is a binary tree

Applications:arithmetic expressionsdecision processessearching

A

B C

F GD E

H I


Arithmetic Expression TreeBinary tree associated with an arithmetic expression

internal nodes: operatorsexternal nodes: operands

Example: arithmetic expression tree for the expression (2 × (a − 1) + (3 × b))

+

××

−2

a 1

3 b


Decision TreeBinary tree associated with a decision process

internal nodes: questions with yes/no answerexternal nodes: decisions

Example: dining decision

Want a fast meal?

How about coffee? On expense account?

Starbucks In ‘N Out Antoine's Denny’s

Yes No

Yes No Yes No

5


Properties of Binary TreesNotationn number of nodese number of

external nodesi number of internal

nodesh height

Properties:e = i + 1n = 2e − 1h ≤ ih ≤ (n − 1)/2e ≤ 2h

h ≥ log2 eh ≥ log2 (n + 1) − 1


Inorder TraversalIn an inorder traversal a node is visited after its left subtree and before its right subtreeApplication: draw a binary tree

x(v) = inorder rank of vy(v) = depth of v

Algorithm inOrder(v)if isInternal (v)

inOrder (leftChild (v))visit(v)if isInternal (v)

inOrder (rightChild (v))

3

1

2

5

6

7 9

8

4


Euler Tour TraversalGeneric traversal of a binary treeIncludes a special cases the preorder, postorder and inorder traversalsWalk around the tree and visit each node three times:

on the left (preorder)from below (inorder)on the right (postorder)

+

×

−2

5 1

3 2

LB

R×


Printing Arithmetic ExpressionsSpecialization of an inorder traversal

print operand or operator when visiting nodeprint “(” before traversing left subtreeprint “)” after traversing right subtree

Algorithm printExpression(v)if isInternal (v)

print(“(’’)inOrder (leftChild (v))

print(v.element ())if isInternal (v)

inOrder (rightChild (v))print (“)’’)

+

××

−2

a 1

3 b((2 × (a − 1)) + (3 × b))


∅

Linked Data Structure for Representing Trees (§6.4.3)

A node is represented by an object storing

ElementParent nodeSequence of children nodes

Node objects implement the Position ADT

B

DA

C E

F

B

∅ ∅

A D F

∅

C

∅

EElementary Data Structures 30

Linked Data Structure for Binary Trees (§6.4.2)

A node is represented by an object storing

ElementParent nodeLeft child nodeRight child node

Node objects implement the Position ADT

B

DA

C E

∅ ∅

∅ ∅ ∅ ∅

B

A D

C E

∅

6


Array-Based Representation of Binary Trees (§6.4.1)

nodes are stored in an array

…

let rank(node) be defined as follows:rank(root) = 1if node is the left child of parent(node),

rank(node) = 2*rank(parent(node))if node is the right child of parent(node),

rank(node) = 2*rank(parent(node))+1

1

2 3

6 74 5

10 11

A

HG

FE

D

C

B

J

1

Introduction to complexity

CSM 1220 - Complexity 2

Complexity: a measure of the performance of an algorithm

An algorithm’s performance depends on internal and external factors

External• Size of the input to the

algorithm• Speed of the computer

on which it is run• Quality of the compiler

InternalThe algorithm’s efficiency, in terms of:• Time required to run• Space (memory storage)

required to run

Complexity measures the internal factors(usually more interested in time than space)


Growth rates and big-O notation

• Growth rates capture the essence of an algorithm’s performance• Big-O notation indicates the growth rate. It is the class of

mathematical formula that best describes an algorithm’s performance, and is discovered by looking inside the algorithm

• Big-O is a function with parameter N, where N is usually the size of the input to the algorithm– For example, if an algorithm depending on the value n has

performance an2 + bn + c (for constants a, b, c) then we say the algorithm has performance O(N2)

• For large N, the N2 term dominates. Only the dominant term is included in big-O

2


Common growth rates

The Towers of Hanoi problemO(2N) exponentialSimultaneous linear equationsO(N3) cubic

Shortest path between two nodes in a graph

O(N2) quadraticSorting n items by ‘divide-and-conquer’O(N log N) n-log-nFinding an entry in an unsorted arrayO(N) linearFinding an entry in a sorted arrayO(log N) logAdding to the front of a linked listO(1) constantExampleTime complexity


Growth rates

Number of Inputs

Tim

e

O(N2)

O(Nlog N)

For a short time N2 isbetter than NlogN


Calculating the actual time taken by a program (example)

• A program takes 10ms to process one data item (i.e. to do one operation on the data item)

• How long would the program take to process 1000 data items, if time is proportional to:– log10 N– N– N log10 N– N2

– N3

• (time for 1 item) x (big-O( ) time complexity of N items)

3


• In some cases, it is important to consider the best, worst and/or average (or typical) performance of an algorithm:

• E.g., when sorting a list into order, if it is already in order then the algorithm may have very little work to do

• The worst-case analysis gives a bound for all possible input (and may be easier to calculate than the average case)

Best, average, worst-case complexity

– Worst, O(N) or o(N): ≥ or > true function *– Typical, Θ(N): ≅ true function *– Best, Ω(N): ≤ true function *

* These approximations are true only after N has passed some value


How do we calculate big-O?

1 Loops2 Nested loops3 Consecutive

statements4 If-then-else statements5 Logarithmic complexity

Five guidelines for finding out the time complexity of a piece of code


Guideline 1: Loops

The running time of a loop is, at most, the running time of the statements inside the loop (including tests) multiplied by the number of iterations.

for (i=1; i<=n; i++)

m = m + 2;

constant timeexecutedn times

Total time = a constant c * n = cn = O(N)

4


Guideline 2: Nested loops

Analyse inside out. Total running time is the product of the sizes of all the loops.

for (i=1; i<=n; i++) for (j=1; j<=n; j++)

k = k+1;

constant time

outer loopexecutedn times

inner loopexecutedn times

Total time = c * n * n * = cn2 = O(N2)


Guideline 3: Consecutive statements

Add the time complexities of each statement.

Total time = c0 + c1n + c2n2 = O(N2)

x = x +1;for (i=1; i<=n; i++)

m = m + 2;for (i=1; i<=n; i++)

for (j=1; j<=n; j++) k = k+1;

inner loopexecutedn times

outer loopexecutedn times constant time

executedn timesconstant time

constant time


Guideline 4: If-then-else statements

Worst-case running time: the test, plus either the then part or the else part (whichever is the larger).

if (depth( ) != otherStack.depth( ) ) return false;

else

for (int n = 0; n < depth( ); n++) if (!list[n].equals(otherStack.list[n]))

return false;

then part:constant

else part:(constant +constant) * n

test:constant

another if :constant + constant(no else part)

Total time = c0 + c1 + (c2 + c3) * n = O(N)

5


Guideline 5: Logarithmic complexity

An algorithm is O(log N) if it takes a constant time to cut the problem size by a fraction (usually by ½)

Example algorithm (binary search):finding a word in a dictionary of n pages

• Look at the centre point in the dictionary• Is word to left or right of centre?• Repeat process with left or right part of

dictionary until the word is found


Performance isn’t everything!

• There can be a tradeoff between:– Ease of understanding, writing and debugging– Efficient use of time and space

• So, maximum performance is not always desirable

• However, it is still useful to compare the performance of different algorithms, even if the optimal algorithm may not be adopted

1

Binary Search Trees

•A dictionary stores a dynamically changing set S and supports the following operations:•Insert(x): Insert element x into S•Delete(x): Removes element x from S•Find(x): Decides whether x is in S and, if so, reports x•If the elements in S can be compared, the following operations are also desirable:•Successor(x) / Predecessor(x): Reports the next larger / smaller element from x in S•Minimum / Maximum: Reports the smallest / largest element in S•Range-Search(x, y): Reports all elements z in S such thatx ≤ z ≤ y

The Dictionary Data Structure

A binary search tree is a binary tree with the following binary search tree property:

Binary Search Tree Property

For any node v, the elements stored in the left subtree of v are not greater than the element stored at v; the elements in the right subtree are not less than the element stored at v.

1

5

13

7

17

18

19

30

37

51

14

25 34

2

The Find Operation — Searching for an Element

Goal: Identify the node storing a given element x, if such a node exists.

1

5

13

7

17

18

19

30

37

51

14

25 34

x = 25


Find(x, T)1 v ← root(T)2 while v ≠ nil and x ≠ key(v)3 do if x < key(v)4 then v ← left_child(v)5 else v ← right_child(v)6 return v


1

5

13

7

17

18

19

30

37

51

14

25 34

x = 25


Lemma: If element x is stored in T, procedure Find finds the node storing x.


1

5

13

7

17

18

19

30

37

51

14

25 34

x = 25

Lemma: The runningtime of procedure Find is O(height(T)).

3

The Minimum Operation — Finding the Smallest Element

Goal: Find the smallest element in S.

1

5

13

7

17

18

19

30

37

51

14

25 34



1

5

13

7

17

18

19

30

37

51

14

25 34

13

17

18

19

30

37

51

14

25 34



1

5

13

7

17

18

19

30

37

51

14

25 34

13

17

18

19

30

37

51

14

25 34

4



1

5

13

7

17

18

19

30

37

51

14

25 34

Minimum(T)1 v ← root(T)2 if v ≠ nil3 then return Subtree-Minimum(v)4 else return nil

Subtree-Minimum(v)1 while left_child(v) ≠ nil2 do v ← left_child(v)3 return v

Lemma: The running timeof procedure Minimum is O(height(T)).

The Successor Operation — Finding the Next Element

Successor(v)1 if right_child(v) ≠ nil2 then return Subtree-Minimum(right_child(v))3 w ← parent(v)4 while w ≠ nil and v = right(w)5 do v ← w6 w ← parent(w)7 return w

Goal: Given an element x S, find the next larger element y S.

1

5

13

7

17

18

19

30

37

51

14

25 34

The Successor Operation — Finding the Next Element

Successor(v)1 if right_child(v) ≠ nil2 then return Subtree-Minimum(right_child(v))3 w ← parent(v)4 while w ≠ nil and v = right(w)5 do v ← w6 w ← parent(w)7 return w

Goal: Given an element x S, find the next larger element y S.

Lemma: The running time of procedure Successor is O(height(T)).

5

The Insert Operation — Inserting a New Element

Goal: Insert a new element xinto the dictionary.

1

5

13

7

17

18

19

30

37

51

14

25 34


Insert(x, T)1 v ← root(T)2 w ← nil3 while v ≠ nil4 do w ← v5 if x < key(v)6 then v ← left_child(v)7 else v ← right_child(v)8 u ← new node with key x9 if w = nil10 then root(T) ← u11 else parent(u) ← w12 if x < key(w)13 then left_child(w) ← u14 else right_child(w) ← u


1

5

13

7

17

18

19

30

37

51

14

25 349


Insert(x, T)1 v ← root(T)2 w ← nil3 while v ≠ nil4 do w ← v5 if x < key(v)6 then v ← left_child(v)7 else v ← right_child(v)8 u ← new node with key x9 if w = nil10 then root(T) ← u11 else parent(u) ← w12 if x < key(w)13 then left_child(w) ← u14 else right_child(w) ← u


Lemma: The running time of procedure Insert is O(height(T)).

Lemma: Procedure Insert preserves the binary search tree property.

6

The Delete Operation — Deleting Element

1

5

13

7

17

18

19

30

37

51

14

25 34

Goal: Delete an element xfrom the dictionary.


1

5

13

7

17

18

19

30

37

51

14

25 34


Delete(v, T)1 if left_child(v) = nil or right_child(v) = nil2 then w ← v3 else w ← Successor(v)4 if left_child(w) ≠ nil5 then u ← left_child(w)6 else u ← right_child(w)7 if u ≠ nil8 then parent(u) ← parent(w)9 if parent(w) = nil10 then root(T) ← u11 else if w = left_child(parent(w))12 then left_child(parent(w)) ← u13 else right_child(parent(w)) ← u14 if w ≠ v15 then key(v) ← key(w)16 Delete node w



1

5

14

7

17

18

19

30

37

51

25 34

7



Lemma: The running time of procedure Delete is O(height(T)).

Lemma: Procedure Delete preserves the binary search tree property.

Delete(v, T)1 if left_child(v) = nil or right_child(v) = nil2 then w ← v3 else w ← Successor(v)4 if left_child(w) ≠ nil5 then u ← left_child(w)6 else u ← right_child(w)7 if u ≠ nil8 then parent(u) ← parent(w)9 if parent(w) = nil10 then root(T) ← u11 else if w = left_child(parent(w))12 then left_child(parent(w)) ← u13 else right_child(parent(w)) ← u14 if w ≠ v15 then key(v) ← key(w)16 Delete node w

Theorem: A binary search tree supports operations Insert, Delete, Find, Minimum, Maximum, Predecessor, and Successor in O(height(T)) time.

Binary Search Tree Operations—Summary

Best case:Tree is completeHeight is Θ(lg n)

Worst case:Tree is a linked list

Height is Θ(n)

111

AVL Trees(See extra AVL notes too)

Cx21120 - Other Trees 2

AVL Trees• A type of binary search tree which is nearly as good as a

balanced tree for time complexity of the operations, and whose structure we can maintain as insertions and deletions proceed, is the AVL tree

• AVL trees are named after the Russian mathematiciansG.M. Adelson-Velskii and E.M. Landis, who discovered them in 1962

• An AVL tree is a binary search tree in which the heights of the left and right subtrees of the root differ by at most 1, and in which the left and right subtrees of the root are again AVL trees


Examples of AVL Trees

Legal AVL, but not balanced

AVL

Non-Legal AVL

AVL and balanced

222


Implementing AVL Trees• To implement algorithms for inserting and deleting from

an AVL tree, we associate a balance factor with each node

• This is left high, right high or equal depending on whether the left subtree of the node has height greater than, less than or equal to that of the right subtree

• Thus the node may be represented by:public class AVLNode

Comparable data;int balanceFactor;AVLNode left;AVLNode right;

public static final int LEFT_HIGH = -1;public static final int RIGHT_HIGH = 1;public static final int EQUAL_HIGH = 0;


Examples of AVL Trees with Balance Factors

L

ER

E

R

R

R

RR

R

E

E R

E E E(AVL trees can be quite skewed)

At L, the left subtree is higher than the rightAt R, the right subtree is higher than the leftAt E, the subtrees are equal (in fact, they are empty)


A Non-AVL Tree (height difference is more than one in

places)E

LL

L

E

R

E

RR

LL means that the left subtree is too highRR means that the right subtree is too high

* Note that no AVL treewill ever have two doubleimbalances, like this, at the same time - we’llsee why later. This is to illustrate how a tree canbe unbalanced somewhereother than the root

333


Insertion into an AVL Tree (the simple case)

Consider constructing a binary search tree in the usual way from the sequence of keys 4, 7, 2, 8, 6, 1, 5, making each node with its balance factor as we go:

4:E 4:R7:E

4:E2:E 7:E

4:R2:E 7:R

8:E

4:R2:E 7:E

6:E 8:E


Insertion contd (inserting 1, 5)

4:E

2:L 7:E

1:E 6:E 8:E

4:R

2:L 7:L1:E

6:L 8:E

5:E

This example shows how the balance factors change as new nodes are added - notice the changes are only on the path from new node to root node


The Insertion Algorithm (for more complex cases)

• Insertion starts as for a binary search tree• The only case in which we need to do anything special is

if the new node is added to a subtree of the root that is higher than the other subtree and the new node increases the height of the subtree

• In this case we have to carry out some balancing operations in the neighbourhood of the root

• We consider the case of inserting into the right subtree(left is similar)

• There are two cases to consider:

444


1 right hand subtree of root is right high after insertion

4:R2:E 7:E

6:E

9:E

RR8:E

Inserting 9 causes 4:R -> 4:RR

Perform a left rotation

7:E4:E 8:R

2:E 6:E9:E

The nodes 2 4 7 8 9,which comprise the‘outside edge’ of thetree, have been rotatedone node to the leftThe 6 has been relocated

as a child of 4.


2 right hand subtree of root is left high after insertion

4:R

2:E 7:E

5

8:E

L

6:E

Inserting 5

Perform a double rotation

• right rotation of right subtree(5 6 7 8 in this example)

• tree now like previous slide• left rotation of whole tree, as before6:E

4:E 7:R2:E 5:E 8:E

left hand child of right hand child of root becomes rootleft hand child of new root becomes right hand child of old root

1

2


The balancing rule:• When you have found the node that is out of balance:

– If that node and the two nodes immediately below it are in a straight line then a single rotation is needed

– If the nodes lie in a dog-leg pattern, then you need a double rotation to restore the balance

• Although the algorithm seems complicated, it still gives insertion in O(log n) time

• Note: there is no ‘LR’ or ‘RL’ imbalance, only ‘LL’ and ‘RR’. The ‘LL’ means the subtree to the left of the node is too deep (the same with ‘RR’ by symmetry). It does not refer to the pattern of the imbalance at all!

555


Another example

From "An introduction to data structures & algorithms with Java”, G.W. Rowe Build an AVL tree using the following data: 10, 20, 30, 25, 27, 7, 4

10:E 10:R

20:E

10:RR

20:R

30:E

The problem lies on a straight line, so apply single rotation


Another example (contd.)

10:E

20:E

30:E

AVL structure is now restored

After inserting 25 and 27, node at 30 becomes unbalanced

10:E

20:

30:LL

25:R

27:E


Rebalance by a double rotation

First part of double rotation rotates 27 to replace 25

10:E

20:

30:LL

27:L

25:E

Second part of rotation moves 27 up to replace 30

10:E

20:R

27:E

25:E 30:E

666


Inserting 7, 4

Inserting 7 causes no problem

10:L

20:E

27:E

25:E 30:E7:E

Inserting 4 causes a problem - it’s a straight-line single rotation

4:E

10:LL

20:

27:E

25:E 30:E7:L


Rebalance by a single rotation

Everything is nicely balanced again

7:E

20:E

27:E

25:E 30:E4:E 10:E


Applications of Binary Trees• Sorting: We can sort data by reading it in, item by item, and

constructing a binary search tree as we go• When we have read all the data, we output it by doing in-order

traversal of the tree• If there is any possibility that the data items are nearly sorted

already, or nearly in reverse order, then it is important to use an AVL tree– Otherwise you end up with a very unbalanced tree!

• The time complexity of this method of sorting is, in general, O(nlog2n+ n) i.e., O(nlog n)

777


Searching maintaining an order• Consider the problem of counting how many times each word

appears in a piece of text (a concordance):– Read the text and build a binary search tree in which the key is a

word (order is alphabetical) and the data is the number of times it occurs

– As we meet each word, we check to see whether it's already in the tree

– If it is, we add one to the data field; otherwise we insert a new node with the word as key and with the data field set to one

– (No need to use an AVL tree because the words will not occur in alphabetical order)

– When we've finished, an in-order traversal allows us to print the words, with their frequencies, in alphabetical order


Compiling arithmetic expressions

• We can represent an arithmetic expression such as(A + B) * (C + D) * 2 – X / Y

as a binary tree, in which the leaves are constants or variables and the nodes are operations:

–/*

X Y2*++

CB DA

A post order traversal then gives us the order in which the operations have to be carried out

1

3

2

4 56

1

Elementary Graph Algorithms

Paths and Cycles

A path is a sequence of vertices P = (v0, v1, …, vk) such that, for 1 ≤ i ≤ k, edge (vi – 1, vi) E.

Path P is simple if no vertex appears more than once in P.

A cycle is a sequence of vertices C = (v0, v1, …, vk – 1) such that, for 0 ≤ i < k, edge(vi, v(i + 1) mod k) E.

Cycle C is simple if the path(v0, v1, vk – 1) is simple.

g

f

i

d

e

b

ah

j

c

Paths and Cycles

g

f

i

d

e

b

ah

j

c





P1 = (g, d, e, b, d, a, h) is not simple.

P2 = (f, i, c, j) is simple.

2

Paths and Cycles





g

f

i

d

e

b

ah

j

c

C1 = (a, h, j, c, i, f, g, d) is simple.

Paths and Cycles





g

f

i

d

e

b

ah

j

c

C1 = (a, h, j, c, i, f, g, d) is simple.

C2 = (g, d, b, h, j, c, i, e, b) is not simple.

A graph H = (W, F) is a subgraph of a graph G = (V, E) if W V and F E.

Subgraphs

3

A graph H = (W, F) is a subgraph of a graph G = (V, E) if W V and F E.

Subgraphs

Spanning Graphs

A spanning graph of G is a subgraph of G that contains all vertices of G.

Spanning Graphs

A spanning graph of G is a subgraph of G that contains all vertices of G.

4

Connectivity

A graph G is connected if there is a path between any two vertices in G.

The connected components of a graph are its maximal connected subgraphs.

Trees

A tree is a graph that is connected and contains no cycles.

Spanning Trees

A spanning tree of a graph is a spanning graph that is a tree.

5

Adjacency List Representation of a Graph

•List of vertices•Pointer to head of adjacency list•List of edges•Pointers to adjacency list entries•Adjacency list entry•Pointer to edge•Pointers to endpoints

v

wx

yu

zab

c

de

wy u x z v

da bc e

•Goal:•Visit all the vertices in the graph of G•Count them•Number them•…

•Identify the connected components of G•Compute a spanning tree of G

Graph Exploration

ExploreGraph(G)1 Label every vertex and edge of G as unexplored2 for every vertex v of G3 do if v is unexplored4 then mark v as visited5 S ← Adj(v)6 while S is not empty7 do remove an edge (u, w) from S8 if (u, w) is unexplored9 then if w is unexplored10 then mark edge (u, w) as tree edge11 mark vertex w as visited12 S ← S Adj(w)13 else mark edge (u, w) as a non-tree edge

Graph Exploration

6

Theorem: Procedure ExploreGraph visits every vertex of G.

Theorem: Every iteration of the for-loop that does not immediately terminate completely explores a connected component of G.

Theorem: The graph defined by the tree edges is a spanning forest of G.

Properties of ExploreGraph

ConnectedComponents(G)1 Label every vertex and edge of G as unexplored2 c ← 03 for every vertex v of G4 do if v is unexplored5 then c ← c + 16 assign component label c to v7 S ← Adj(v)8 while S is not empty9 do remove an edge (u, w) from S10 if (u, w) is unexplored11 then if w is unexplored12 then mark edge (u, w) as tree edge13 assign component label c to w14 S ← S Adj(w)15 else mark edge (u, w) as a non-tree edge

Connected Components

BFS(G)1 Label every vertex and edge of G as unexplored2 for every vertex v of G3 do if v is unexplored4 then mark v as visited5 S ← Adj(v)6 while S is not empty7 do (u, w) ← Dequeue(S)8 if (u, w) is unexplored9 then if w is unexplored10 then mark edge (u, w) as tree edge11 mark vertex w as visited12 Enqueue(S, Adj(w))13 else mark edge (u, w) as a non-tree edge

Breadth-First Search

Running time: O(n + m)

7

Properties of Breadth-First Search

Theorem: Breadth-first search visits the vertices of G by increasing distance from the root.Theorem: For every edge (v, w) in G such that v Li andw Lj, |i – j| ≤ 1.

L0

L1 L2

L3

L4

DFS(G)1 Label every vertex and edge of G as unexplored2 for every vertex v of G3 do if v is unexplored4 then mark v as visited5 S ← Adj(v)6 while S is not empty7 do (u, w) ← Pop(S)8 if (u, w) is unexplored9 then if w is unexplored10 then mark edge (u, w) as tree edge11 mark vertex w as visited12 Push(S, Adj(w))13 else mark edge (u, w) as a non-tree edge

Depth-First Search

Running time: O(n + m)

An Important Property of Depth-First Search

Theorem: For every non-tree edge (v, w) in a depth-first spanning tree of an undirected graph, v is an ancestor of w or vice versa.

8

A Recursive Depth-First Search Algorithm

DFS(G)1 Label every vertex and every edge of G as unexplored2 for every vertex v of G3 do if v is unexplored4 then DFS(G, v)

DFS(G, v)1 mark vertex v as visited2 for every edge (v, w) in Adj(v)3 do if (v, w) is unexplored4 then if w is unexplored5 then mark edge (v, w) as tree edge6 DFS(G, w)7 else mark edge (v, w) as a non-tree edge

Documents

Algorithms and Data Structures - vowi.fsinf.at¼hrung_in_C(plusplus... · Algorithms and Data Structures Markus Würzl, Vienna University of Technology Solving Interesting Problems