PARALLEL AGORITHMS

ABILITIES AND LIMITATIONS

MURTADHA AL-SABBAGH

PARALLEL ALGORITHMS ABILITES

THROGHPUT AND LATENCY• Throughput: Is the number of operations done per time

unit.

• Latency : Is the time needed to complete one operation.

EXAMPLE (THROUGHPUT AND LATENCY)Suppose we have two vehicles a car that can carry two persons with an average speed of 200 km/h and a bus that can carry 40 persons with a speed of 50 km/h , find the latency and throughput if they both used to carry a group of people for a distance of 4500 km ?

Car

• Latency =22.5

• Throughput=0.089

Bus

• Latency =90

• Throughput=0.45

THROUGHPUT AND LATENCY• We can use parallelism to increase throughput by using

a larger number of lower clocked processing units (as in the GPU) which is well suited for computation intensive applications ( applications with need of large number of calculations such as image processing applications).

• Also we can use it in latency sensitive applications which are applications with need of fast response (e.g. Windows applications which many parts running together at the same time). In this case we use a more powerful processing units such as (Intel core i7).

• Throughput and latency often contradictory .

WHY USE PARALLEL COMPUTING?

1) The Real World is Massively Parallel:

In the natural world, many complex, interrelated events are happening at the same time, yet within a temporal sequence.

Compared to serial computing, parallel computing is much better suited for modeling, simulating and understanding complex, real world phenomena.

For example, imagine modeling these serially:

WHY USE PARALLEL COMPUTING?

2)SAVE TIME AND/OR MONEY:• In theory, throwing more resources at a task will shorten its time to

completion, with potential cost savings.• Parallel computers can be built from cheap, commodity

components.

3) SOLVE LARGER / MORE COMPLEX PROBLEMS:

• Many problems are so large and/or complex that it is impractical or impossible to solve them on a single computer, especially given limited computer memory.

• Example: Web search engines/databases processing millions of transactions per second

• Using compute resources on a wide area network, or even the Internet when local compute resources are scarce or insufficient.

4) TAKE ADVANTAGE OF NON-LOCAL RESOURCES:

PARALLEL ALGORITHMS LIMITS

THE LIMITATIONS

We Face the following limitations when designing a parallel program:

1. Amadahl’s law.

2. Complexity.

3. Portability.

4. Resource Requirements.

5. Scalability.

6. Parallel Slowdown

AMDAHL’S LAW

Amdahl’s law provides an upper bound on the speedup that can be obtained by a parallel program

Let us suppose that a processor need an ‘n’ time units to complete ‘n’ operations .

And we have a program that have a ‘p’ part of parallizable code and ‘1-p’ or ‘s’ un-paralleizable part of code.

Then the processing time for a ‘N’ number of processors is :

=

And hence the speed up ratio will be calculated as a ratio between a single processor time ‘Ts’ and an ‘N’ processors processing time ‘Tp’.

AMDAHL’S LAW

Introducing the number of processors performing the parallel fraction of work, the relationship can be modeled by:

N = number of processors and S = serial fraction

AMDAHL’S LAW

If we consider N is very large then :

COMPLEXITY

The costs of complexity are measured in every aspect of the software development cycle:

1. Design

2. Coding

• Data dependency

• Race conditions

3. Debugging

4. Tuning

5. Maintenance

DATA DEPENDENCYResults from multiple use of the same location(s) in storage by different tasks.

e.g.

for (int i=0;i<100;i++)

array[i]=array[i-1]*20;

How to Handle Data Dependencies:

• Distributed memory architectures - communicate required data at synchronization points.

• Shared memory architectures -synchronize read/write operations between tasks.

RACE CONDITION

Thread A Thread B

1A: Read variable V 1B: Read variable V

2A: Add 1 to variable V 2B: Add 1 to variable V

3A: Write back to variable V 3B: Write back to variable V

If instruction 1B is executed between 1A and 3A, or if instruction 1A is executed between 1B and 3B, the program will produce incorrect data. This is known as a race condition .

SYCNCHRONIZATION• Our program often requires "serialization" of segments of

the program.

• Synchronization is the solution for the above problems.

• To implement synchronization we use:

• Barrier .

• Locks .

• Synchronous communication operations .

 PORTABILITY

Although there is several standardization APIs, such as MPI, POSIX threads, and OpenMP, portability issues with parallel programs are not as serious as in years past. However...

Operating systems can play a key role in code portability issues.

Hardware architectures are characteristically highly variable and can affect portability.

RESOURCE REQUIREMENTS• The primary intent of parallel programming is to decrease

execution wall clock time, however in order to accomplish this, more CPU time is required. For example, a parallel code that runs in 1 hour on 8 processors actually uses 8 hours of CPU time.

• The amount of memory required can be greater for parallel codes than serial codes, due to the need to replicate data and for overheads associated with parallel support libraries and subsystems.

• For short running parallel programs, there can actually be a decrease in performance compared to a similar serial implementation. The overhead costs associated with setting up the parallel environment, task creation, communications and task termination can comprise a significant portion of the total execution time for short runs.

SCALABILITY

Two types of scaling based on time to solution:

• Strong scaling: The total problem size stays fixed as more processors are added.

• Weak scaling: The problem size per processor stays fixed as more processors are added.

Hardware factors play a significant role in scalability. Examples:

• Memory-CPU bus bandwidth • Amount of memory available on any given machine or set

of machines• Processor clock speed

PARALLEL SLOWDOWN

• Not all parallelization results in speed-up.

• Once task split up into multiple threads those threads spend a large amount of time communicating among each other resulting degradation in the system.

• This is known as parallel slowdown.

THANK YOU^_~