UPC-CHECK Project Final ReportHigh Performance Computing Group

Iowa State UniversityAug 30, 2011

This is the final report for contract DE-AC05-00OR22726, subcontract modification 4000078135, modification 4.

Deliverables included:Sections 2.1.A –B, regarding OpenShem Performance and Validation Tests were descoped, and so are not included. Section 2.2A OpenSHMEM-CHECK design document was delivered in Dec 2010, and is included in the Final Report distributed.

Sections 2.2.B1-4 Relate to the implementation of UPC-CHECK using ROSE, and were delivered in Jun 2011. This I will discuss further. A half-day tutorial is scheduled for Aug 30, 14:15-17:00.

Outline

• Description of UPC-CHECK• Ease of Use• Ability to Detect Errors• Speed/Scalability• Memory Overhead• Installation• Summary

Description of UPC-CHECK

• UPC-CHECK is a tool for detecting run-time errors in Unified Parallel C (UPC) programs.

• This is done by translating a UPC program to another UPC program with run-time checks.

• ISU Project was for Deadlock Detection and errors in UPC Function Arguments.

• Translator was implemented via the LLNL Rose toolkit.

5

UPC-CHECK Design

Original UPC Program

UPC to UPCTranslator

UPC program with error checking

UPC-CHECK Support Routines

UPC Compiler

Executable with error checking

Ease of Use• UPC-CHECK is designed to be easy to use:• On the command line replace upcc –o a.exe program.upc sub.o –l lib with upc-check –o a.exe program.upc sub.o –l lib

• In Makefiles, just replace UPCC=upccwith UPCC=upc-check

7

Scalability of UPC-CHECK checks

Type of check Overhead (for T threads)

Argument checking O(1)

Deadlocks Collective routines O(1) UPC_Locks O(L) , L<=T

Where L is the length of the longest hold-and-wait chain. For a program that does not use upc_locks, the overhead in using UPC-CHECK does not depend on the number of threads. This is because all checking can be done via values local to the threads and its neighboring threads. The O(1) deadlock checking for collective routines will be described in a paper that is being prepared.

A program that uses upc_locks may have overhead that depends on the number of threads because there may be a chain of lock dependencies (a deadlock) which spans all threads.

8

Memory Overhead

The memory overhead per thread consists of three components:

1) Extra context variables allocated to support checks: approximately 128KB.2) Extra information about call stack (--upccheck:-s option) 1/2 KB per call level3) Executable size: Support routines: less than 1MB Each UPC routine adds about 3.5Kbytes.

Installation• UPC-CHECK is based on LLNL’s ROSE which uses

BOOST and the EDG compiler for UPC support.• Installation of BOOST and ROSE with EDG support

is the first step. See http://rosecompiler.org• UPC-CHECK is installed via the command: install_UPC-CHECK• By default, BOOST, ROSE and UPC-CHECK are

in /usr/local, options allow installation elsewhere.

• Assumes a UNIX/LINUX environment.

Run-Time Errors Detected by UPC-CHECK

• UPC-CHECK detects Argument Errors in UPC Functions and finds Deadlocks in UPC programs.

• UPC-CHECK will not test the single-valued requirement of upc_forall statements.

• Since UPC-CHECK works on UPC source programs, it cannot detect deadlocks within library functions.

• Currently, UPC-CHECK requires that programs do not define the ‘main' function in a header file.

Quantifying the quality of a tool which detects UPC run-time errors.

• Iowa State University has a Test Suite that scores the ability of UPC compilers/tools to detect run-time errors: see http://rted.public.iastate.edu/UPC/

• This Test Suite uses the following scoring system:– A score of 5 is given for a detailed error message that will assist a

programmer to quickly correct the error.– A score of 4 is given for error messages with more information than a score

of 3 and less than 5. – A score of 3 is given for error messages with the correct error name, line

number and the name of the file where the error occurred.– A score of 2 is given for error messages with the correct error name and line

number where the error occurred but not the file name where the error occurred.

– A score of 1 is given for error messages with the correct error name.– A score of 0 is given when the error was not detected.

How UPC-CHECK compares• Results from ISU’s test suite:http://rted.public.iastate.edu/UPC/RESULTS/result_table.html

• UPC-CHECK gets the highest score for Deadlocks and the highest score for all but 3 tests in the Argument Errors section.

Compiler Argument Errors DeadlocksUPC-CHECK 4.89 5.00Berkley UPC 0.04 0.58Cray 0.38 0.00HP 0.00 0.36GNU 0.00 0.27

The reason for 3 tests not scoring a 5 in the Argument Errors.

• Three tests in the Argument Errors section get a zero score.

• These tests violate the single-valuedness requirement of the upc_forall.

• UPC-CHECK does not perform this check. • Detecting this would have required that UPC-

CHECK serialize all controlling upc_forall loops in the program.

Additional Checks

• While collecting the information necessary to instrument the Argument Errors and Deadlock checks, UPC-CHECK sometimes detects a different error. Whichever error would occur first is reported with a meaningful error message. (E.g. a collective routine within a upc_forall.)

• Due to this, more categories in ISU’s RTED_UPC test suite showed improvement when using UPC-CHECK, see:

http://rted.public.iastate.edu/UPC/RESULTS/result_table.html

• In addition, some errors are detected and reported at translation/compile time.

Execution Overhead Example• UPC-CHECK has been designed to minimize the overhead when executing the instrumented UPC program. • We timed the UPC implementation of the NAS Parallel CG benchmark with/without UPC-CHECK using Berkley UPC.• S, A, and B are the smallest benchmarks, so they will show the most overhead. • We see essentially zero overhead using UPC-CHECK. WallTime(sec.), 4 ThreadsSize No UPC-CHECK With UPC-CHECK Overhead S 7.36 7.41 0.7% A 9.06 9.12 0.7% B 85.03 83.04 - 2.3%

Overhead on a Cray XT using the Cray compiler 128 threads

NAS Benchmark Execution Time for Original Program (sec.)

Execution Time for Instrumented Program (sec.) Slowdown

CG-A4.912 4.99 1.02

CG-B54.183 54.239 1.00

CG-C58.309 58.281 1.00

EP-A1.417 1.427 1.01

EP-B7.116 7.128 1.00

EP-C11.19 11.17 1.00

IS-A3.56 3.658 1.03

IS-B8.752 8.776 1.00

IS-C10.089 10.073 1.00

Total159.528 159.742 1.00

SummaryUPC-CHECK:• Detects and reports run-time errors in UPC programs

involving: UPC Function Arguments and Deadlocks,• Provides high-quality error messages,• Is easy to use,• Has low run-time overhead,• Is scalable, and• Is available at: http://hpcgroup.public.iastate.edu/UPC-CHECK