Parallel Programming with PVM Prof. Sivarama Dandamudi School of Computer Science Carleton University

Parallel Programming with PVM

Prof. Sivarama DandamudiSchool of Computer Science

Carleton University

Carleton University © S. Dandamudi 2

Parallel Algorithm Models Five basic models

Data parallel modelTask graph modelWork pool modelMaster-slave modelPipeline modelHybrid models


Parallel Algorithm Models (cont’d)

Data parallel modelOne of the simplest of all the modelsTasks are statically mapped onto processorsEach task performs similar operation on different data

Called data parallelism modelWork may be done in phases

Operations in different phases may be differentEx: Matrix multiplication



Data parallel modelA11 A12 B11 B12 C11 C12A21 A22 B21 B22 C21 C22A decomposition into four tasks

Task 1: C11 = A11 B11 + A12 B21Task 2: C12 = A11 B12 + A12 B22Task 3: C21 = A21 B11 + A22 B21Task 4: C11 = A21 B21 + A22 B22

. =



Task graph modelParallel algorithm is viewed as a task-dependency

graphCalled task parallelism model

Typically used for tasks that have large amount of dataStatic mapping is used to optimize data movement cost

Locality-based mapping is importantEx: Divide-and-conquer algorithms, parallel quicksort



Task parallelism



Work pool modelDynamic mapping of tasks onto processors

Important for load balancingUsed on message passing systems

When the data associated with a task is relatively smallGranularity of tasks

Too small: overhead in accessing tasks can increaseToo big: Load imbalance

Ex: Parallelization of loops by chunk scheduling



Master-slave modelOne or more master processes generate work and

allocate it to worker processesAlso called manager-worker model

Suitable for both shared-memory and message passing systems

Master can potentially become a bottleneckGranularity of tasks is important



Pipeline modelA stream of data passes through a series of processors

Each process performs some task on the dataAlso called stream parallelism model

Uses producer-consumer relationshipOverlapped executionUseful in applications such as database query processing

Potential problem One process can delay the whole pipeline



Pipeline model

R5

R4

R3

R2R1

Pipelined processing canavoid writing temporaryresults on disk and reading them back



Hybrid modelsPossible to use multiple modelsHierarchical

Different models at different levelsSequentially

Different models in different phasesEx: Major computation may use task graph model

Each node of the graph may use data parallelism or pipeline model


PVM Parallel virtual machine

Collaborative effortOak Ridge National Lab, University of Tennessee,

Emory University, and Carnegie Mellon UniversityBegan in 1989

Version 1.0 was used internallyVersion 2.0 released in March 1991Version 3.0 in February 1993


PVM (cont’d)

Parallel virtual machineTargeted for heterogeneous network computing

Different architecturesData formatsComputational speedsMachine loadsNetwork load


PVM Calls Process control

int tid = pvm_mytid(void)Returns pid of the calling processCan be called multiple times

int info = pvm_exit(void)Does not kill the processTells the local pvmd that this process is leaving PVMinfo < 0 indicates error (error: pvmd not responding)


PVM Calls (cont’d)

Process controlint numt = pvm_spawn(char *task, char **argv, int flag, char *where, int ntask, int *tids)

Starts ntask copies of the executable file task

Arguments to task (NULL terminated)

Specific host

Specific architecture(PVM_ARCH)



flag specifies options

Value Option Meaning

0 PvmTaskDefault PVM chooses where to span

1 PvmTaskHost where specifies a host

2 PvmTaskArch where specifies a architecture

3 PvmTaskDebug Starts tasks under debugger



Process controlint info = pvm_kill(int tid)

Kills the PVM task identified by tidDoes not kill the calling taskTo kill the calling task

First call pvm_exit()Then exit()

Writes to the file /tmp/pvml.<uid>



Informationint tid = pvm_parent(void)

Returns the tid of the process that spawned the calling task

Returns PvmNoParent if the task is not created by pvm_spawn()



Informationint info = pvm_config( int *nhost, int *narch, struct pvmhostinfo **hostp)

Returns nhost = number of hosts Returns narch = number of different data formats



Message sending Involves three steps

Send buffer must be initialized Use pvm_initsend()

Message must be packed Use pvm_pk*() Several pack routines are available

Send the message Use pvm_send()



Message sendingint bufid = pvm_initsend( int encoding)

Called before packing a new message into the bufferClears the send buffer and creates a new one for

packing a new message bufid = new buffer id



encoding can have three options: PvmDataDefault

XDR encoding is used by default Useful for heterogeneous architectures

PvmDataRaw No encoding is done Messages sent in their original form

PvmDataInPLace No buffer copying Buffer should not be modified until sent



Packing data Several routines are available (one for each data type) Each takes three arguments

int info = pvm_pkbyte(char *cp, int nitem, int stride)

nitem = # items to be packed stride = stride in elements



Packing datapvm_pkint pvm_pklongpvm_pkfloat pvm_pkdoublepvm_pkshort

Pack string routine requires only the NULL-terminated string pointer

pvm_pkstr(char *cp)



Sending dataint info = pvm_send(int tid, int msgtag)

Sends the message in the packed buffer to task tidMessage is tagged with msgtag

Message tags are useful to distinguish different types of messages



Sending data (multicast)int info = pvm_mcast(int *tids, int ntask, int msgtag)

Sends the message in the packed buffer to all tasks in the tid array (except itself)

tid array length is given by ntask



Receiving dataTwo steps

Receive dataUnpack it

Two versionsBlocking

Waits until the message arrivesNon-blocking

Does not wait



Receiving dataBlocking receiveint info = pvm_recv(int tid, int msgtag)

Wait until a message with msgtag has arrived from task tid

Wildcard value (1) is allowed for both msgtag and tid



Receiving dataNon-blocking receive

int info = pvm_nrecv(int tid, int msgtag)

If no message with msgtag has arrived from task tidReturns bufid = 0

Otherwise, behaves like the blocking receive



Receiving dataProbing for a messageint info = pvm_probe(int tid, int msgtag)

If no message with msgtag has arrived from task tidReturns bufid = 0

Otherwise, returns a bufid for the messageDoes not receive the message



Unpacking data (similar to packing routines)pvm_upkint pvm_upklongpvm_upkfloat pvm_upkdoublepvm_upkshort pvm_upkbyte

Pack string routine requires only the NULL-terminated string pointer

pvm_upkstr(char *cp)



Buffer informationUseful to find the size of the received message

int info = pvm_bufinfo(int bufid, int *bytes, int *msgtag, int *tid)

Returns msgtag, source tid, and size in bytes


Example Finds sum of elements of a given vector

Vector size is given as inputThe program can be run on a PVM with up to 10 nodes

Can be modified by changing a constantVector is assumes to be evenly divisable by number of

nodes in PVM Easy to modify this restriction

Master (vecsum.c) and slave (vecsum_slave.c) programs


Example (cont’d)

vecsum.c#include <stdio.h>#include <sys/time.h>#include "pvm3.h"

#define MAX_SIZE 250000 /* max. vector size */#define NPROCS 10 /* max. number of PVM nodes */


Example (cont’d)

main(){int cc, tid[NPROCS];long vector[MAX_SIZE];double sum = 0,

partial_sum; /* partial sum received from slaves */long i, vector_size;


Example (cont’d)

int nhost, /* actual # of hosts in PVM */ size; /* size of vector to be distributed */

struct timeval start_time, finish_time;long sum_time;


Example (cont’d)

printf("Vector size = ");scanf("%ld", &vector_size);

for(i=0; i<vector_size; i++) /* initialize vector */ vector[i] = i;

gettimeofday(&start_time, (struct timezone*)0); /* start time */


Example (cont’d)

tid[0] = pvm_mytid(); /* establish my tid */

/* get # of hosts using pvm_config() */pvm_config(&nhost, (int *)0,

(struct hostinfo *)0);

size = vector_size/nhost; /* size of vector to send to slaves */


Example (cont’d)

if (nhost > 1) pvm_spawn("vecsum_slave", (char **)0,

0, "", nhost-1, &tid[1]);for (i=1; i<nhost;i++){

/* distribute data to slaves */ pvm_initsend(PvmDataDefault); pvm_pklong(&vector[i*size],size,1); pvm_send(tid[i],1);}


Example (cont’d)

for (i=0; i<size;i++) /* perform local sum */ sum += vector[i];for (i=1; i<nhost;i++){

/* collect partial sums from slaves */ pvm_recv(-1,2); pvm_upkdouble(&partial_sum,1,1); sum += partial_sum;}


Example (cont’d)

gettimeofday(&finish_time, (struct timezone*)0); /* finish time */

sum_time = (finish_time.tv_sec – start_time.tv_sec) * 1000000 + finish_time.tv_usec -

start_time.tv_usec;

Time in secs


Example (cont’d)

printf("Sum = %lf\n",sum);printf("Sum time on %d hosts =

%lf sec\n", nhost, (double)sum_time/1000000);pvm_exit();

}


Example (cont’d)

vecsum_slave.c#include "pvm3.h"#define MAX_SIZE 250000main(){int ptid, bufid, vector_bytes;long vector[MAX_SIZE];double sum = 0;int i;


Example (cont’d)

ptid = pvm_parent(); /* find parent tid */bufid = pvm_recv(ptid,1);

/* receive data from master *//* use pvm_bufinfo() to find the number of bytes received */pvm_bufinfo(bufid, &vector_bytes,

(int *)0, (int *) 0);


Example (cont’d)

pvm_upklong(vector, vector_bytes/sizeof(long),1); /* unpack */for (i=0;

i<vector_bytes/sizeof(long); i++) /* local summation */ sum += vector[i];


Example (cont’d)

pvm_initsend(PvmDataDefault); /* send sum to master */pvm_pkdouble(&sum,1,1);pvm_send(ptid, 2);

/* use msg type 2 for partial sum */

pvm_exit();}

Documents

Parallel Programming with PVM Prof. Sivarama Dandamudi School of Computer Science Carleton University