View
214
Download
0
Tags:
Embed Size (px)
Citation preview
1
Dynamic Verification of Multicore Communication Applications in MCAPI
Subodh Sharma, UofUGanesh Gopalakrishnan, UofU
Eric Mercer, BYU
1Supported by NSF CCF 0903408/0903491 and SRC 2009-TJ-1993/1994
HLDVT 2009
2
Concurrency Space in Multicore Era
S/W solution H/W solution
Cloud/HPC Computing MPI, PVM, Mapreduce, MSCS
Clusters, Vector machines, Supercomputers
Desktop computing OOSD, Scripting, Pthreads, TBB, CT, OpenMP
Desktop machines
Embedded Computing Lightweight counterparts of the above.
FPGAs, DoC, SoC, etc.
3
Concurrency Space in Multicore Era
Requires Standardization
Multicore Association (MCA) Effort
S/W solution H/W solution
Cloud/HPC Computing MPI, PVM, Mapreduce, MSCS
Clusters, Vector machines, Supercomputers
Server/Desktop computing
OOSD, Scripting, Pthreads, TBB, CT, OpenMP
Desktop machines
Embedded Computing Lightweight counterparts of the above.
FPGAs, DoC, SoC, etc.
4
What is MCAPI (Multicore Communication API)?
• An API specification from MCA (Multicore Association)
• To program embedded systems like mobile phones, PDAs, routers, servers, etc.
• Not restricted to SPMD (like MPI) or multi threaded style of programming.
5
Taxonomy for MCA APIs
MCAPI – Communication API• Message based• Packet/Scalar Channel based
MRAPI – Resource Management API
• Semaphores, mutexes• Shared memory segment allocation, deallocation
MTAPI – Task Management API• Specification work yet to begin• Thread Pooling, Work Stealing queues e.g. CILK, TBB, TPL etc.Inter API
interactions
6
State of MCA Efforts?
• MCAPI standard is released.
• MCAPI reference implementation exists• A commercial implementation – Polycore, Inc.• In-house implementation on FPGA in progress (Utah).
• MRAPI standard work in progress.
• MTAPI efforts yet to begin.
7
State of MCA Efforts?
• Standardization efforts take a long time to mature
• Need for FV in MCA API development.
• Our effort: Early Engagement of FV in the MCA APIs space
• To promote early adoption of the API• To promote better programming practices• To help avoid early pit-falls – e.g. unspecified behaviors
8
Achievements so far!
• Formal Specification work (at BYU)• Permits symbolic analysis
• Murphi -> CVC• Query Oracle for scenario outcome calculation.• Platform testing
• Dynamic Verification of MCAPI user apps (at Utah)• MCC tool created.• Current capabilities of MCC
• Dynamic search of dependencies• Exerting a deterministic runtime control
• Algorithmic Advances• Symmetry reduction in MCAPI domain space.
9
Focus of the talk!
• Formal Specification work (at BYU)• Permits execution based symbolic analysis
• Murphi -> CVC• Query Oracle for scenario outcome calculation.• Platform testing
• Dynamic Verification of MCAPI user apps (at Utah)• MCC tool created.• Current capabilities of MCC
• Dynamic search of dependencies• Exerting a deterministic runtime control
• Algorithmic Advances• Symmetry reduction in MCAPI domain space.
10
Related Work
• CHESS tool from MSR• Uses preemption bounded search • Cannot be used for message passing programs
• ISP tool from Utah• Used for MPI verification• MCC differs in the way determinism is enforced at runtime.
• Inspect tool from Utah• Used for shared memory programs• Implements the classical DPOR algorithm.
• JPF• A model checker for Java bytecode.
11
MCAPI Preliminaries
• MCAPI Node – • Processes, Threads, OS instance, Processor core,
etc.
• MCAPI Endpoint –• Socket-like communication termination points• Identified by a tuple <node_id, port_id>
• MCAPI Channels • Point to point unidirectional FIFO connections
between a pair of endpoints.
• A single communication universe
12
MCAPI Messages
Node 1
Port 1 <1,1>
Port 2 <1,2>
Node 2
Port 1 <2,1>
Connectionless Message
To <2,1>
• Message received at FIFO receive queue
13
MCAPI Connection Oriented Communication
Node 1
Port 1 <1,1>
Port 2 <1,2>
Node 2
Port 1 <2,1>To <2,1>
• Sent over a connected channel• <1,1> can communicate on ONE channel only
Node 3
Port 1 <3,1>
To <3,1>ERROR!!
14
MCAPI API generic + message calls
Initialize/Finalize Sets and deletes the environment. Must be called by each communicating node.
Create/Delete endpoint API calls to manage creation/deletion of endpoints on the owner node
Get_endpoint A blocking call to get the address of a remote endpoint.
Send (send_endpoint, recv_endpoint .. )
Sends a data-payload from a sending endpoint to a receiving endpoint. API provides blocking and non blocking versions.
Recv (recv_endpoint, ..) Receives a data payload from the receiving endpoint. API provides blocking non-blocking versions.
Wait (request_handle, ..), Test(req_handle, ..)
Checks the completion of the request. Wait – Blocking; Test – Non blocking
15
Illustration of MCAPI C program
int main () {…for(t=0; t<NUM_THREADS; t++){ rc = pthread_create(&threads[t], NULL, run_thread, (void*)&thread_data_array[t]); } for (t = 0; t < NUM_THREADS; t++) { pthread_join(threads[t],NULL); }}void* run_thread (void *t) {… mcapi_initialize(tid,&version,&status); if (tid == 1) { recv_endpt = mcapi_create_endpoint (PORT_NUM, &status);
mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE,&recv_size, &status);mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE, &recv_size, &status);} else {send_endpt = mcapi_create_endpoint (PORT_NUM,&status);recv_endpt = mcapi_get_endpoint (1,PORT_NUM,&status);mcapi_msg_send(send_endpt,recv_endpt, msg,strlen(msg), 1,&status);} mcapi_finalize(&status);return NULL;}
Potential deadlock
16
MCC – MCAPI Checker
Workflow of MCC.
MCAPI C ProgramMCAPI C Program
instrumentation
Instrumented ProgramInstrumented Program
compile
MCAPI Library WrapperMCAPI Library Wrapper
Scheduler
Executable
thread 1
thread n
request/permit
request/
permit
17
Bug Classes in MCAPI User Applications
• Non determinism due to communication race.• The recv() call is passed with only receiving
endpoint as a parameter. • The receiving endpoint extracts message from the
FIFO receive queue.
T0 T1 T2
--- --- --- The send that matches the recv may decide the correctness !!
POTENTIAL BUG
S(e1,e2) S(e1,e2) R(e2)
R(e2)
18
Bug Classes in MCAPI User Applications
• Mismatched Calls
“get_endpoint” invoked for a non-existent endpoint -- ERROR
T0 T1
--- ---
R(1) Get_endpoint(2)
S(1,2)
Does not exist
19
Challenge: exponential interleavings
19
T0 T1 T2 T3 T4 T5
T0 T1 T2 T3 T4 T5
A B
Dependent MCAPI calls
Only these 2 are RELEVANT!!!
Possible Interleaving
20
Relevant interleaving exploration by MCC scheduler
T0 T1 T2
S (ep1,ep3) S(ep2,ep3) R(ep3)
R(ep3)
Intra happens-Before Edge
21
T0 T1 T2
S (ep1,ep3) S (ep2,ep3) R (ep3)
R (ep3)
Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3)
Match Set: <S(ep1, ep3), R (ep3)> <S(ep2, ep3), R (ep3)>
Interleaving 1
Issue to the runtime
Match sets are computed at “decision points”
Set of matchingtransitions
No runnable thread
Relevant interleaving exploration by MCC scheduler
22
T0 T1 T2
S (ep1,ep3) S (ep2,ep3) R (ep3)
R (ep3)
Enabled Transitions: S(ep2, ep3), R (ep3)
Match Set: <S(ep2, ep3), R (ep3)>
Interleaving 1
Issue to the runtime
Relevant interleaving exploration by MCC scheduler
23
T0 T1 T2
S (ep1,ep3) S (ep2,ep3) R (ep3)
R (ep3)
Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3)
Match Set: <S(ep2, ep3), R (ep3)>
Interleaving 2
Issue to the runtime
Relevant interleaving exploration by MCC scheduler
24
T0 T1 T2
S (ep1,ep3) S (ep2,ep3) R (ep3)
R (ep3)
Enabled Transitions: S(ep2, ep3), R (ep3)
Match Set: <S(ep1, ep3), R (ep3)>
Interleaving 2
Issue to the runtime
Relevant interleaving exploration by MCC scheduler
25
T0 T1 T2
S (ep1,ep3) S (ep2,ep3) R (ep3)
R (ep3)
Enabled Transitions: S(ep2, ep3), R (ep3)
Match Set: <S(ep1, ep3), R (ep3)>
Interleaving 2
Issue to the runtimeError Caught
Relevant interleaving exploration by MCC scheduler
26
Enforcing a deterministic runtime match• For non blocking requests, there is an additional problem
• Runtime communication race even after a scheduler decides deterministically.
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Match-set 1
Match-set 2
27
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Match-set 1
Match-set 2
Not finished yet!!
RACE
Enforcing a deterministic runtime match
28
Solution 1 to enforce a deterministic match.
• Addition of Probing function to the MCAPI library• Probes an endpoint’s receive queue for a message• Returns TRUE if the message is found
• Scheduler issues the next match-set only when the probe function returns TRUE.
Enforcing a deterministic runtime match
29
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Probing Solution
Receive Queue
matched
Enforcing a deterministic runtime match
30
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Probing Solution
Receive Queue
matched
Runtime MCC Scheduler
Probes
Enforcing a deterministic runtime match
31
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Probing Solution
T1 Receive Queue
Probes
Runtime MCC Scheduler
Enforcing a deterministic runtime match
32
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Probing Solution
T1T0 Receive Queue
Enforcing a deterministic runtime match
33
Solution 2 to enforce a deterministic match.
• Scheduler induces a “test” call
• Scheduler spin-loops on the request handle until the
successful completion of the “test” call.
We opt Solution 2 in our work as it is non-intrusive.
Enforcing a deterministic runtime match
34
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Enabled Transitions: Send(ep1, ep3), Send(ep2, ep3), Recv (ep3)
Match Set: <Send(ep2, ep3), Recv (ep3)> <Send(ep1, ep3), Recv (ep3)>
Dummy “wait” Solution
matched
Enforcing a deterministic runtime match
35
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Dummy “wait” Solution
Runtime MCC Scheduler
Enforcing a deterministic runtime match
Test ()
36
T0 T1 T2
IS (ep1,ep3) IS (ep2,ep3) IR (ep3)
IR (ep3)
Dummy “wait” Solution
Runtime MCC Scheduler
Enforcing a deterministic runtime match
Test()
matched
37
Concluding remarks and future work
• MCC currently supports connectionless API constructs• Checks for safety assertion violations and
Deadlocks
• Steadily improve MCC over this year• Support for connection-oriented API calls, sanity
checks etc.
• Release MCC and assess experience of industrial app writers
39
Happens-Before Relationship – Computed Dynamically
T0 T1 T2
IS (ep1,ep3, r1) IS (ep2,ep3, r3) IR (ep3, r4)
IR (ep3, r5)IS (ep1,ep4, r2)
R (ep3)W(r2)
40
t1: a[x] := …
t2: … := a[y] t1: a[x] := …
t2: … := a[y]
Equal ??
t2: … := a[y]
Review of POR
Dynamic POR
Dependencies based on concrete values
x=56, y=753 (e.g.) at runtime!
41
Happens-Before Relationship – Example
T0 T1 T2
IS (ep1,ep3, r1) IS (ep2,ep3, r3) IR (ep3, r4)
IR (ep4, r5)IS (ep1,ep4, r2)
R (ep3)W(r2)