39
Always MPI_TEST or MPI_WAIT Jeff Squyres The MPI Guy December 2012

Friends don't let friends leak MPI_Requests

Embed Size (px)

DESCRIPTION

How leaking MPI MPI_Requests hurts your application. Always be sure to use an MPI TEST or WAIT operation on your pending MPI_Requests!

Citation preview

Page 1: Friends don't let friends leak MPI_Requests

Always MPI_TEST or MPI_WAITJeff SquyresThe MPI Guy

December 2012

Page 2: Friends don't let friends leak MPI_Requests

Isn’t this a correct MPI application?

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

Page 3: Friends don't let friends leak MPI_Requests

Isn’t this a correct MPI application?

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}NO!

Page 4: Friends don't let friends leak MPI_Requests

Isn’t this a correct MPI application?

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

Must (eventually) use an MPI TEST or WAIT operation on the pending request, even if you “know” that it has already completed

Page 5: Friends don't let friends leak MPI_Requests

But

WHY?

Page 6: Friends don't let friends leak MPI_Requests

A. Because MPI says so.

Page 7: Friends don't let friends leak MPI_Requests

But I don’t WANT TO!

Page 8: Friends don't let friends leak MPI_Requests

A. Because MPI says so.

Don’t fight the Law, son.The Law will always win.

Page 9: Friends don't let friends leak MPI_Requests

B. Because you’ll leak resources

Page 10: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

Page 11: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a request

isend_request

Page 12: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a requestMPI performs the send

isend_request

Page 13: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a requestMPI performs the send…which may include allocating additional resources

isend_request

resources

Page 14: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

MPI allocates a requestMPI performs the send…which may include allocating additional resourcesThose resources are attached to the request

isend_request

resources

Page 15: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

The send eventually done

isend_request

resources

Page 16: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

The send eventually finishesThe request is marked complete

isend_request

resourcesALL DONE!

Page 17: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

…but it’s still there!

isend_request

resourcesALL DONE!

Page 18: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

…but it’s still there!Eating valuable memory!

isend_request

resourcesALL DONE!

Page 19: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);

…but it’s still there!Eating valuable memory!And possibly still holding on to valuable resources!

isend_request

resourcesALL DONE!

Page 20: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);MPI_Wait(&request, …);

…until you invoke a TESTor WAIT operation thatcompletes the request

isend_request

resourcesALL DONE!

Page 21: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);MPI_Wait(&request, …);

Then the request is freed resources

Page 22: Friends don't let friends leak MPI_Requests

Consider what happens when you Isend…

MPI_Isend(buffer, …, &request);MPI_Wait(&request, …);

Then the request is freed…including any resources thatweren’t previously released

Page 23: Friends don't let friends leak MPI_Requests

Does it really

MATTER?

Page 24: Friends don't let friends leak MPI_Requests

YES

Page 25: Friends don't let friends leak MPI_Requests

Individual MPI_Requestsmay be smallOne tiny

request

Page 26: Friends don't let friends leak MPI_Requests

All of RAM

Particularly compared tohow much RAM is availableOne tiny

request

Page 27: Friends don't let friends leak MPI_Requests

All of RAM

…but every MPI processis making requests

Page 28: Friends don't let friends leak MPI_Requests

All of RAM

…and they might be makingLOTS of requests

Page 29: Friends don't let friends leak MPI_Requests

All of RAM

You might fill RAM slowly…but you’ll still fill it

Page 30: Friends don't let friends leak MPI_Requests

Worse: hardware resources might notbe released until the request is complete

isend_request

Remember these? resources

Page 31: Friends don't let friends leak MPI_Requests

Meaning: it’s not just abundantRAM that you’re consuming

resources

Page 32: Friends don't let friends leak MPI_Requests

Communication hardware = expensiveTherefore, far less resources available

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

resources

Page 33: Friends don't let friends leak MPI_Requests

Compare: RAM vs. communication hardware resources

All of RAM

Page 34: Friends don't let friends leak MPI_Requests

Compare: RAM vs. communication hardware resources

All of RAMWasting this may be forgivable

Page 35: Friends don't let friends leak MPI_Requests

Compare: RAM vs. communication hardware resources

All of RAMWasting this may be forgivable

Wasting this isa terrible idea

Page 36: Friends don't let friends leak MPI_Requests

THIS is a correct MPI application

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …); MPI_Wait(&request, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

Page 37: Friends don't let friends leak MPI_Requests

THIS is a correct MPI application

void sender(void) { MPI_Request request; int ack; /* … */ MPI_Isend(buffer, …, &request); MPI_Recv(&ack, …); MPI_Wait(&request, …);}

void receiver(void) { int ack; /* … */ MPI_Recv(buffer, …); MPI_Send(&ack, …);}

If you “know” the send is already complete, the WAITwill likely be fast

Page 38: Friends don't let friends leak MPI_Requests

Friends don’t let friendsleak MPI_Requests

Page 39: Friends don't let friends leak MPI_Requests

Thank you.