1

An Approach to Formalization and Analysis of Message Passing Libraries

Robert Palmer Intel Validation Research Labs, Hillsboro, OR

(work done at the Univ of Utah as PhD student)

Michael DeLisi(undergraduate; his first research paper)

Ganesh GopalakrishnanRobert M. Kirby

School of ComputingUniversity of Utah

Supported by: Microsoft HPC Institutes

NSF CNS 0509379

2

MPI is the de-facto standard for programming cluster machines

Our focus: Eliminate Concurrency Bugs from HPC Programs !

An Inconvenient Truth: Bugs More CO2 , Bad Numbers !

(BlueGene/L - Image courtesy of IBM / LLNL)(Image courtesy of Steve Parker, CSAFE, Utah)

3

So many ways to eliminate bugs …

Our Contribution:

• A Formal Model of 50 (of the 300) MPI functions• An execution environment from which to check simple “litmus tests”• VisualStudio Integration with Microsoft Phoenix Compiler Front-end• Has spawned other research (e.g. develop POR for MPI programs, and In-Situ model checker for MPI Programs)• Formalization helped reveal omissions in standard• Can potentially help designer understand today’s complex standards• Recommended for future libraries (APIs)

4

A Simple MPI /C Program

/* Add-up integrals calculated by each process */

if (my_rank == 0) {

total = integral;

for (source = 0; source < p; source++) {

MPI_Recv(&integral, 1, MPI_FLOAT,source,

tag, MPI_COMM_WORLD, &status);

total = total + integral;

}

} else {

MPI_Send(&integral, 1, MPI_FLOAT, dest,

tag, MPI_COMM_WORLD);

}

04/18/23

5

Library Semantics Dictates Behaviore.g. mismatched send/recv causing deadlock

/* Add-up integrals calculated by each process */

if (my_rank == 0) {

total = integral;

for (source = 0; source < p; source++) {

MPI_Recv(&integral, 1, MPI_FLOAT,source,

tag, MPI_COMM_WORLD, &status);

total = total + integral;

}

} else {

MPI_Send(&integral, 1, MPI_FLOAT, dest,

tag, MPI_COMM_WORLD);

}

04/18/23

p1:to 0 p2:to 0 p3:to 0

p0:fr 0 p0:fr 1 p0:fr 2

6

Challenges for SW Model Checking

04/18/23

• Build Debugging Tools that “understand” Library Semantics

• Perform Static Analysis and Model Reductions modulo Library Semantics!

A new world-order where the embedding program serves as a ‘control scaffolding’ with the “action” being within library calls

7

Library Semantics Modeling Approaches

04/18/23

• Natural Language Documents - They alone don’t suffice (obvious drawbacks)

• Formal Descriptions - Use standard notations, ideally executable

8

Practitioners must be able to benefit…

04/18/23

1. They must be able to gain deeper understanding of the library thru the spec

2. Must be able to submit “litmus tests” and see outcomes in familiar ways

9

Retain the Level of Detail of Interest

04/18/23

10

Example: Challenge posed by a 5-line MPI program…

04/18/23

p0: { Irecv(rcvbuf1, from p1); Irecv(rcvbuf2, from p1); … }

p1: { sendbuf1 = 6; sendbuf2 = 7; Issend(sendbuf1, to p0); Isend (sendbuf2, to p0); … }

• In-order message delivery (rcvbuf1 == 6)

• Can access the buffers only after a later wait / test

• The second receive may complete before the first

• When Issend (synch.) is posted, all that is guaranteed is that Irecv(rcvbuf1,…) has been posted

11

Our Contributions

04/18/23

1. Formal Executable Spec of the point-to-point operations of MPI – written in TLA+

2. Simple MPI / C programs are compiled into TLA+ models and linked with Formal Semantics – all under Microsoft VisualStudio

3. Errors in Litmus Tests generate error traces that can step the Visual-Studio debugger

4. Same Framework includes a customized MPI model checker and soon a Dynamic Execution-based Model Checker with DPOR

12

One of our Litmus Tests

13

Executable Formal Specification and MPIC Model Checker Integration into VS

04/18/23

TLA+ MPI Library Model

TLA+ Prog. Model

MPIC Program Model

Visual Studio 2005

Phoenix Compiler

TLC Model Checker MPIC Model Checker

Verification Environment

MPIC IR

14

MPI Formal Specification Organization

04/18/23

MPI 1.1 API

Point to Point Operations

Collective Operations

Requests

Communicator

Collective

Context Group

Constants

15

The Histrionics of FV for HPC (1)

16

The Histrionics of FV for HPC (2)

17

Error-trace Visualization in VisualStudio

18

Spec of MPI_Wait (Slide 1 of 2)

19

Spec of MPI_Wait (Slide 2 of 2)

20

Related Work (Formalization and tool integration)

1. Use of TLA+ (or similar notations) to write executable specs is nothing new

2. Use to model a subset of MPI is new

3. Integration with VS and VS-debugger (or similar tools) may help designers become comfortable with formal specs

21

Related Work (MPI formalization)

Siegel (VMCAI 2007) has proposed a Promela model for MPI

Uses Promela constructions to mimic MPI behavior

Uses the Promela / C interface

Uses an elaborately hand-crafted state machineIs much faster, and rides on established technology

The declarative reading (emphasizing “what”) is lost

22

Control state machine used in Siegel (VMCAI 2007)

23

Concluding Remarks (1 of 3)

Quote from Lynn Conway (quote in VLSI, paraphrased):

“There are two realities that must be met –

- the architecture of a microprocessor, and

- the polygons of the layout.

Everything in-between is a luxury to be availed depending on our resources.”

24

Concluding Remarks (2 of 3)

In our world:

- The executable formal spec: the “what” (architecture)

- An In-Situ Dynamic Partial Order Reduction (ISP)

model checker is the “how” (or “polygons”) (paper

in EuroPVM / MPI 2007)

- The “in-between” is a customized MPI model checker

for some of its constructs (PADTAD 2007)

25

Concluding Remarks (3 of 3)

– Formal Spec of Concurrency Libraries is essential for the development of a whole range of FV tools

– It helps programmers avoid misunderstandings about the library

– It can help during the platform testing of Library Implementations (think about multicores and transactions used in future library implementations

The Model Checking Community and the Formal Spec

Community must work hand-in-hand in addressing

the issues in tomorrow’s Parallel and Distributed Programs

26

Partial Demo

27

Questions ?

The verification environment is downloadable from

http://www.cs.utah.edu/formal_verification/mpic

It is at an early stage of development

28

Answers!1. We are extending it to Collective Operations

- lesson learned from de Supinski

2. We may perform Formal Testing of MPI Library Implementations based on the Formal Semantics

3. We plan to analyze mixed MPI / Threads

4. That is a very good question – let’s talk!