Lecture 5 MPI Introduction Topics …Readings January 19, 2012 CSCE 713 Advanced Computer Architecture

Lecture 5MPI Introduction

Lecture 5MPI Introduction

Topics Topics …

ReadingsReadings

January 19, 2012

CSCE 713 Advanced Computer Architecture

– 2 –CSCE 713 Spring 2012

A distributed memory systemA distributed memory system

Copyright © 2010, Elsevier Inc. All rights Reserved


A shared memory systemA shared memory system



Identifying MPI processesIdentifying MPI processes

Common practice to identify processes by nonnegative Common practice to identify processes by nonnegative integer ranks.integer ranks.

pp processes are numbered processes are numbered 0, 1, 2, .. p-10, 1, 2, .. p-1



Our first MPI programOur first MPI program



CompilationCompilation

mpicc -g -Wall -o mpi_hello mpi_hello.c


ExecutionExecution


mpiexec -n <number of processes> <executable>

mpiexec -n 1 ./mpi_hello


run with 1 process

run with 4 processes


ExecutionExecution




Greetings from process 0 of 1 !

Greetings from process 0 of 4 !Greetings from process 1 of 4 !Greetings from process 2 of 4 !Greetings from process 3 of 4 !


MPI ProgramsMPI ProgramsWritten in C.Written in C.

Has main. Uses stdio.h, string.h, etc.

Need to add Need to add mpi.hmpi.h header file. header file.

Identifiers defined by MPI start with “MPI_”.Identifiers defined by MPI start with “MPI_”.

First letter following underscore is uppercase.First letter following underscore is uppercase. For function names and MPI-defined types. Helps to avoid confusion.


– 10 –CSCE 713 Spring 2012

MPI ComponentsMPI Components

MPI_InitMPI_Init Tells MPI to do all the necessary setup.

MPI_FinalizeMPI_Finalize Tells MPI we’re done, so clean up anything allocated for this

program.


– 11 –CSCE 713 Spring 2012

Basic OutlineBasic Outline


– 12 –CSCE 713 Spring 2012

Communicators Communicators A collection of processes that can send messages to A collection of processes that can send messages to

each other.each other.

MPI_Init defines a communicator that consists of all the MPI_Init defines a communicator that consists of all the processes created when the program is started.processes created when the program is started.

Called Called MPI_COMM_WORLDMPI_COMM_WORLD..


– 13 –CSCE 713 Spring 2012

CommunicatorsCommunicators


number of processes in the communicator

my rank (the process making this call)

– 14 –CSCE 713 Spring 2012

SPMDSPMD

Single-Program Multiple-DataSingle-Program Multiple-Data

We compile We compile oneone program. program.

Process 0 does something different.Process 0 does something different. Receives messages and prints them while the other

processes do the work.

The The if-elseif-else construct makes our program SPMD. construct makes our program SPMD.


– 15 –CSCE 713 Spring 2012

CommunicationCommunication


– 16 –CSCE 713 Spring 2012

Data typesData types


– 17 –CSCE 713 Spring 2012

CommunicationCommunication


– 18 –CSCE 713 Spring 2012

Message matchingMessage matching


MPI_Sendsrc = q

MPI_Recvdest = r

r

q

– 19 –CSCE 713 Spring 2012

Receiving messagesReceiving messages

A receiver can get a message without knowing:A receiver can get a message without knowing: the amount of data in the message, the sender of the message, or the tag of the message.


– 20 –CSCE 713 Spring 2012

status_p argumentstatus_p argument


MPI_SOURCEMPI_TAG

MPI_ERROR

MPI_Status*

MPI_Status* status;

status.MPI_SOURCEstatus.MPI_TAG

– 21 –CSCE 713 Spring 2012

How much data am I receiving?How much data am I receiving?


– 22 –CSCE 713 Spring 2012

Issues with send and receiveIssues with send and receive

Exact behavior is determined by the MPI Exact behavior is determined by the MPI implementation.implementation.

MPI_Send may behave differently with regard to buffer MPI_Send may behave differently with regard to buffer size, cutoffs and blocking.size, cutoffs and blocking.

MPI_Recv always blocks until a matching message is MPI_Recv always blocks until a matching message is received.received.

Know your implementation;Know your implementation;don’t make assumptions!don’t make assumptions!


– 23 –CSCE 713 Spring 2012

TRAPEZOIDAL RULE IN MPITRAPEZOIDAL RULE IN MPI


– 24 –CSCE 713 Spring 2012

The Trapezoidal RuleThe Trapezoidal Rule


– 25 –CSCE 713 Spring 2012

The Trapezoidal RuleThe Trapezoidal Rule


– 26 –CSCE 713 Spring 2012

One trapezoidOne trapezoid


– 27 –CSCE 713 Spring 2012

Pseudo-code for a serial programPseudo-code for a serial program


– 28 –CSCE 713 Spring 2012

Parallelizing the Trapezoidal RuleParallelizing the Trapezoidal Rule

1.1. Partition problem solution into tasks.Partition problem solution into tasks.

2.2. Identify communication channels between tasks.Identify communication channels between tasks.

3.3. Aggregate tasks into composite tasks.Aggregate tasks into composite tasks.

4.4. Map composite tasks to cores.Map composite tasks to cores.


– 29 –CSCE 713 Spring 2012

Parallel pseudo-codeParallel pseudo-code


– 30 –CSCE 713 Spring 2012

Tasks and communications for Trapezoidal RuleTasks and communications for Trapezoidal Rule


– 31 –CSCE 713 Spring 2012

First version (1)First version (1)


– 32 –CSCE 713 Spring 2012



– 33 –CSCE 713 Spring 2012



– 34 –CSCE 713 Spring 2012

Dealing with I/ODealing with I/O


Each process justprints a message.

– 35 –CSCE 713 Spring 2012

Running with 6 processesRunning with 6 processes


unpredictable output

– 36 –CSCE 713 Spring 2012

Input Input Most MPI implementations only allow process 0 in Most MPI implementations only allow process 0 in

MPI_COMM_WORLD access to MPI_COMM_WORLD access to stdinstdin..

Process 0 must read the data (scanf) and send to the Process 0 must read the data (scanf) and send to the other processes.other processes.


– 37 –CSCE 713 Spring 2012

Function for reading user inputFunction for reading user input


– 38 –CSCE 713 Spring 2012

COLLECTIVE COMMUNICATIONCOLLECTIVE COMMUNICATION


– 39 –CSCE 713 Spring 2012

Tree-structured communicationTree-structured communication1.1. In the first phase: In the first phase:

(a) Process 1 sends to 0, 3 sends to 2, 5 sends to 4, (a) Process 1 sends to 0, 3 sends to 2, 5 sends to 4, and 7 sends to 6. and 7 sends to 6. (b) Processes 0, 2, 4, and 6 add in the received (b) Processes 0, 2, 4, and 6 add in the received values. values. (c) Processes 2 and 6 send their new values to (c) Processes 2 and 6 send their new values to processes 0 and 4, respectively.processes 0 and 4, respectively.(d) Processes 0 and 4 add the received values into (d) Processes 0 and 4 add the received values into their new values.their new values.

2.2. (a) Process 4 sends its newest value to process 0.(a) Process 4 sends its newest value to process 0.(b) Process 0 adds the received value to its newest (b) Process 0 adds the received value to its newest value.value.


– 40 –CSCE 713 Spring 2012

A tree-structured global sumA tree-structured global sum


– 41 –CSCE 713 Spring 2012

OverviewOverviewLast TimeLast Time

Posix Pthreads: create, join, exit, mutexes /class/csce713-006 Code and Data

Readings for todayReadings for today http://csapp.cs.cmu.edu/public/1e/public/ch9-preview.pdf

NewNew Website alive and kicking; dropbox too! From Last Lecture’s slides: Gauss-Seidel Method, Barriers,

Threads Assignment Next time performance evaluation, barriers and MPI intro

– 42 –CSCE 713 Spring 2012

Threads programming AssignmentThreads programming Assignment

1.1. Matrix addition (embarassingly parallel)Matrix addition (embarassingly parallel)

2.2. VersionsVersionsa. Sequential

b. Sequential with blocking factor

c. Sequential Read without conversions

d. Multi threaded passing number of threads as command line argument (args.c code should be distributed as an example)

3.3. Plot of several runsPlot of several runs

4.4. Next timeNext time

– 43 –CSCE 713 Spring 2012

Next few slides are from Parallel Programming by Next few slides are from Parallel Programming by PachecoPacheco

/class/csce713-006/Code/Pacheco contains the code /class/csce713-006/Code/Pacheco contains the code (again only for use with this course do not distribute)(again only for use with this course do not distribute)

ExamplesExamples

1.1. Simple use of mutex in calculating sumSimple use of mutex in calculating sum

2.2. SemaphoreSemaphore

– 44 –CSCE 713 Spring 2012

Global sum function that uses a mutex (2)Global sum function that uses a mutex (2)


– 45 –CSCE 713 Spring 2012

Barriers - synchronizing threadsBarriers - synchronizing threads

Synchronizing threads after a period of computationSynchronizing threads after a period of computation

• No thread proceeds until all others have reached the No thread proceeds until all others have reached the barrierbarrier

E.g., last time iteration of Gauss-Siedel, sync check for E.g., last time iteration of Gauss-Siedel, sync check for convergenceconvergence

Examples of barrier uses and implementationsExamples of barrier uses and implementations

1.1. Using barriers for testing convergence, i.e. satisfying Using barriers for testing convergence, i.e. satisfying a completion criteriaa completion criteria

2.2. Using barriers to time the slowest threadUsing barriers to time the slowest thread

3.3. Using barriers for debuggingUsing barriers for debugging

– 46 –CSCE 713 Spring 2012

Using barriers for testing convergence, i.e. satisfying a completion criteriaUsing barriers for testing convergence, i.e. satisfying a completion criteriaSlide 43 of Lecture 3 (slightly modified)Slide 43 of Lecture 3 (slightly modified)

General computationGeneral computation

Stop when all xStop when all xii from this iteration calculated from this iteration calculated

ii

i

ii

i,

n

j1j

jj,

a

xac

x

n

1i

2i )x- (berror i

– 47 –CSCE 713 Spring 2012

Using barriers to time the slowest threadUsing barriers to time the slowest thread


– 48 –CSCE 713 Spring 2012

Using barriers for debuggingUsing barriers for debugging


– 49 –CSCE 713 Spring 2012

Barrier ImplementationsBarrier Implementations

1.1. Mutex , threadCounter, busy-waitsMutex , threadCounter, busy-waits

2.2. Mutex plus barrier semaphoreMutex plus barrier semaphore

– 50 –CSCE 713 Spring 2012

Busy-waiting and a MutexBusy-waiting and a Mutex

Implementing a barrier using busy-waiting and a mutex Implementing a barrier using busy-waiting and a mutex is straightforward.is straightforward.

We use a shared counter protected by the mutex.We use a shared counter protected by the mutex.

When the counter indicates that every thread has When the counter indicates that every thread has entered the critical section, threads can leave the entered the critical section, threads can leave the critical section.critical section.


– 51 –CSCE 713 Spring 2012

Busy-waiting and a MutexBusy-waiting and a Mutex


We need one counter variable for each

instance of the barrier,

otherwise problemsare likely to occur.

– 52 –CSCE 713 Spring 2012

Implementing a barrier with semaphoresImplementing a barrier with semaphores


– 53 –CSCE 713 Spring 2012

Condition VariablesCondition Variables

A condition variable is a data object that allows a A condition variable is a data object that allows a thread to suspend execution until a certain event or thread to suspend execution until a certain event or condition occurs.condition occurs.

When the event or condition occurs another thread can When the event or condition occurs another thread can signal the thread to “wake up.”signal the thread to “wake up.”

A condition variable is always associated with a mutex.A condition variable is always associated with a mutex.


– 54 –CSCE 713 Spring 2012

Condition VariablesCondition Variables


– 55 –CSCE 713 Spring 2012

Implementing a barrier with condition variablesImplementing a barrier with condition variables


– 56 –CSCE 713 Spring 2012

POSIX Semaphores POSIX Semaphores

$ man -k semaphore$ man -k semaphoresem_close (3) - close a named semaphoresem_close (3) - close a named semaphoresem_destroy (3) - destroy an unnamed semaphoresem_destroy (3) - destroy an unnamed semaphoresem_getvalue (3) - get the value of a semaphoresem_getvalue (3) - get the value of a semaphoresem_init (3) - initialize an unnamed semaphoresem_init (3) - initialize an unnamed semaphoresem_open (3) - initialize and open a named semaphoresem_open (3) - initialize and open a named semaphoresem_overview (7) - Overview of POSIX semaphoressem_overview (7) - Overview of POSIX semaphoressem_post (3) - unlock a semaphoresem_post (3) - unlock a semaphoresem_timedwait (3) - lock a semaphoresem_timedwait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_trywait (3) - lock a semaphoresem_unlink (3) - remove a named semaphoresem_unlink (3) - remove a named semaphoresem_wait (3) - lock a semaphoresem_wait (3) - lock a semaphore

$ man –s 7 semaphores$ man –s 7 semaphoresNo manual entry for semaphore in section 7No manual entry for semaphore in section 7

– 57 –CSCE 713 Spring 2012

Implementing a barrier with condition variablesImplementing a barrier with condition variables


– 58 –CSCE 713 Spring 2012

System V Semaphores (Sets)System V Semaphores (Sets)

In System V IPC there are semaphore setsIn System V IPC there are semaphore setsCommandsCommands• ipcs - provide information on ipc facilities ipcs - provide information on ipc facilities • ipcrm - remove a message queue, semaphore set or shared ipcrm - remove a message queue, semaphore set or shared

memory segment etc.memory segment etc.

System Calls:System Calls:semctl (2) - semaphore control operationssemctl (2) - semaphore control operationssemget (2) - get a semaphore set identifiersemget (2) - get a semaphore set identifiersemop (2) - semaphore operationssemop (2) - semaphore operationssemtimedop (2) - semaphore operationssemtimedop (2) - semaphore operations

– 59 –CSCE 713 Spring 2012

Documents

Lecture 5 MPI Introduction Topics …Readings January 19, 2012 CSCE 713 Advanced Computer Architecture