Friends don't let friends leak MPI_Requests

Always MPI_TEST or MPI_WAITJeff SquyresThe MPI Guy

December 2012

Isn’t this a correct MPI application?

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

Isn’t this a correct MPI application?

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}NO!

Isn’t this a correct MPI application?

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

Must (eventually) use an MPI TEST or WAIT operation on the pending request, even if you “know” that it has already completed

But

WHY?

A. Because MPI says so.

But I don’t WANT TO!

A. Because MPI says so.

Don’t fight the Law, son.The Law will always win.

B. Because you’ll leak resources

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a request

isend_request

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a requestMPI performs the send

isend_request

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a requestMPI performs the send…which may include allocating additional resources

isend_request

resources

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a requestMPI performs the send…which may include allocating additional resourcesThose resources are attached to the request

isend_request

resources

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

The send eventually done

isend_request

resources

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

The send eventually finishesThe request is marked complete

isend_request

resourcesALL DONE!

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

…but it’s still there!

isend_request

resourcesALL DONE!

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

…but it’s still there!Eating valuable memory!

isend_request

resourcesALL DONE!

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

…but it’s still there!Eating valuable memory!And possibly still holding on to valuable resources!

isend_request

resourcesALL DONE!

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);MPI_Wait(&request, …);

…until you invoke a TESTor WAIT operation thatcompletes the request

isend_request

resourcesALL DONE!

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);MPI_Wait(&request, …);

Then the request is freed resources

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);MPI_Wait(&request, …);

Then the request is freed…including any resources thatweren’t previously released

Does it really

MATTER?

YES

Individual MPI_Requestsmay be smallOne tiny

request

All of RAM

Particularly compared tohow much RAM is availableOne tiny

request

All of RAM

…but every MPI processis making requests

All of RAM

…and they might be makingLOTS of requests

All of RAM

You might fill RAM slowly…but you’ll still fill it

Worse: hardware resources might notbe released until the request is complete

isend_request

Remember these? resources

Meaning: it’s not just abundantRAM that you’re consuming

resources

Communication hardware = expensiveTherefore, far less resources available

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

Compare: RAM vs. communication hardware resources

All of RAM

Compare: RAM vs. communication hardware resources

All of RAMWasting this may be forgivable

Compare: RAM vs. communication hardware resources

All of RAMWasting this may be forgivable

Wasting this isa terrible idea

THIS is a correct MPI application

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …); MPI_Wait(&request, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

THIS is a correct MPI application

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …); MPI_Wait(&request, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

If you “know” the send is already complete, the WAITwill likely be fast

Friends don’t let friendsleak MPI_Requests

Thank you.