Testing Concurrent/Distributed Systems

• Testing Concurrent/Distributed Systems

• Review of Final

CEN 5076 Class 14 – 12/05

CEN 5076 Class 14 - 12/05 2

Regression Testing

Regression test approaches:

• Retest All – rerun all baseline test

• Retest Risky Use cases – choose baseline test to rerun by risk heuristics.

• Retest by Profile – choose baseline tests to rerun by allocating time in proportion to the optional profile.

• Retest Changed Segments – choose baseline tests to rerun by comparing code segments.

CEN 5076 Class 14 - 12/05 3

Regression Testing

Regression test approaches:

• Retest Within Firewall – choose baseline tests to rerun by analyzing dependencies.

Points to note:

• Time and cost constraints may preclude running such a test, in which case a reduced regression test can be selected, supporting selective regression testing.

• Reduction of a test suites should use a safe reduction approach.

CEN 5076 Class 14 - 12/05 4

Regression Testing

• A 100% safe regression suite consists of “all test that could possibly exhibit output when run on [the delta build]” [Rothermel ’94].

• Criteria for a safe regression test suite:

1. Inclusiveness – percentage of baseline tests that may show regression faults and that are selected for a reduced test suite.

2. Precision – percentage of baseline test cases in a reduced test suite that cannot reveal regression faults and that are not selected for the reduced test set.

CEN 5076 Class 14 - 12/05 5

Regression Testing

3. Efficiency – the cost to identify a reduced regression test suite.

4. Generality – the range of application for the selection strategy.

• A safe regression test suite is 100 percent inclusive.

• A precise regression test suite does not include any tests that cannot cause different output as these tests can be safely removed.

CEN 5076 Class 14 - 12/05 6

Testing Concurrent/Distributed Systems

• Few systems today are designed to execute using a single process.

• Computational models:

1. Concurrent

2. Parallel

3. Networked

4. Distributed

CEN 5076 Class 14 - 12/05 7

Testing Concurrent Programs

• Concurrent program P consist of several processes,

• P runs on a single processor machine whose process scheduler suspends the running process randomly after any of it atomic instructions, and

• Scheduler randomly chooses a process to run.

Note that this model is equivalent to one in which the instructions are allowed to execute simultaneously. [Weiss S. N. 88]

CEN 5076 Class 14 - 12/05 8

Testing Concurrent Programs • Sequential approach - one program run per

test case. Problem for concurrent programs! Why?

• The execution of a concurrent (centralized, parallel, or distributed) program P exercises a sequence of events called a synchronization sequence or SYN-seq. [Tai K.C. 89]

Give an example of a SYN-seq.

• The result of an execution of P with input X is determined by P, X and the SYN-seq of the execution.

CEN 5076 Class 14 - 12/05 9

Testing Concurrent Programs cont

• Note due to the unpredictable nature of concurrent processes and the use of nondeterministic statements, multiple executions of P with the same input may exercise different SYN-seqs and may produce different results. [Tai K. C. 89]

• How would you go about testing concurrent programs?

CEN 5076 Class 14 - 12/05 10

Testing Concurrent Programs cont

• Two approaches: nondeterministic and deterministic testing.

• Assume the specification for a concurrent program is A, which models valid program behavior. Let P be a concurrent program. Let (P, X) represent program P with input X.

Defn: A SYN-seq S is said to be feasible (valid) for (P, X) if S is allowed by the implementation (specification) of (P, X).

CEN 5076 Class 14 - 12/05 11

Testing Concurrent Programs cont

Defn: P is said to have a synchronization fault if a feasible (valid) SYN-seq of (P, X) is invalid (infeasible).

Nondeterministic testing of P against S:

1. Select a set of test cases

2. For each selected input X execute P one or more times and collect the output and SYN-seq of each execution

3. Determine the validity of each collected SYN-seq

Any problems with nondeterministic testing?

CEN 5076 Class 14 - 12/05 12

Testing Concurrent Programs cont

Problems with deterministic testing:

1. Some feasible SYN-seqs may be executed many times.

2. Some feasible SYN-seqs of (P, X) may never be executed.

3. The existence of valid but infeasible SYN-seqs may never be detected.

Any advantages?

CEN 5076 Class 14 - 12/05 13

Testing Concurrent Programs cont

Deterministic testing of P against S:

1. Select a set of test cases, each of the form (X, S)

2. For each (X, S) perform a “forced execution” of P according to X and S

3. Check results of the forced execution (S is feasible for (P, X) iff the forced execution exercises exactly S before P terminates)

Any problems with deterministic testing?

Any advantages of deterministic testing?

CEN 5076 Class 14 - 12/05 14

Testing Concurrent Programs cont

Problems:

1. Selection of (X, S) pairs

2. Need to modify some component of the implementation e.g., compiler, runtime system, or operating system, to force an execution

Advantages:

1. Allows SYN-seqs to satisfy specific test criterion.

2. Can detect the existence of feasible, invalid SYN-seqs of P as well as valid infeasible SYN-seqs of (P, X).

3. Regression testing is possible after changing P.

CEN 5076 Class 14 - 12/05 15

Testing Concurrent Programs cont

• Nondeterministic testing and deterministic testing are extremes, there are several hybrid between:

1. Reachability testing [Hwang G. H. et al. 95]

2. Prefix-based testing [Tai K. C. et al. 89]

3. Temporal testing [Yang et al. 98]

• Early work in testing concurrent programs focused on debugging. Why?

CEN 5076 Class 14 - 12/05 16

Specification-Based Technique [Carver et al. 98]

• Uses constraints to guide the selection of test sequences.

• Constraints written using CSPE (Constraints on Succeeding and Preceding Events).

• CSPE describes feasibility constraints – sequencing constraints that are satisfied by the feasible SYN-seqs of P.

• Validity constraints are an extension of feasibility constraints – constraints expected to be satisfied by the feasibility SYN-seqs of P.

CEN 5076 Class 14 - 12/05 17

Specification-Based Technique [Carver et al. 98]

S0

Example: n-slot bounded buffer (BB)

S1 S2

deposit deposit

withdraw withdraw

B1. a[#; → deposit]: Buffer BB can always accept deposit first.S0 has an outgoing transition for deposit. B2. ~[#;→ withdraw]: Buffer BB cannot accept withdraw firstS0 has no outgoing transition for withdraw.B3. p[deposit; → deposit] (buffer is not full)

…

“a” – always“~” – never“p” – possibly“#” – start of execution“$” – normal termination“→” – immediately after

CSPE can be formally defined as logical formulas.

CEN 5076 Class 14 - 12/05 18

Specification-Based Technique [Carver et al. 98]

Constraint-based testing methodology:

1. Select a sequencing constraint notation, define how to achieve coverage and detect violations of constraints in the notation, determine constraint coverage criterion.

2. According to P’s spec use the notation to derive a set of validity constraints.

3. Perform nondeterministic testing of P and collect the SYN-seqs . Analyze the SYN-seqs to measure coverage and detect violations of P’s validity constraints.

CEN 5076 Class 14 - 12/05 19

Specification-Based Technique [Carver et al. 98]

Constraint-based testing methodology:

4. Generate additional SYN-seqs of P with the intention of using these SYN-seqs to cover the validity constraints not yet covered.

5. Perform deterministic testing of P with the generated SYN-sequences. (if deterministic testing of P with a generated SYN-seq fails to cover a constraint then a constraint violation is detected.)

6. After corrections have been made to P perform deterministic (regression) testing with the SYN-seqs that previously uncovered violations.

CEN 5076 Class 14 - 12/05 20

Program-Based Technique [Yang et al. 98]

• Supports test data adequacy criteria.

• Automatically generates all-definition-use-paths for parallel programs (really!).

• Program model consists of: threads that communicate through shared variables; syn events – post and wait; and thread creation pthread_create.

• Handles problem of nondeterminism by performing controlled execution through temporal testing i.e.,

CEN 5076 Class 14 - 12/05 21

Program-Based Technique [Yang et al. 98]

• Temporal testing alters the scheduled execution time of a program segment in order to detect syn errors.

• A temporal test case is a 3-tuple (P, X, D), P – program, X – input, D – timing changes. D is the schedule execution time for certain syn instructions n represented as t(n) will change for each temporal test the behavior of P observed.

Example of n - delay points along the du-path being tested i.e., all process creation and syn-events in du-path.

CEN 5076 Class 14 - 12/05 22

Program-Based Technique [Yang et al. 98]

Steps of testing process:

1. Generate all-du-paths statically.

2. Execute the program multiple times without considering any timing changes.

3. Examine the trace results. Different paths executed indicates possibility of syn error.

4. Generate temporal test cases w.r.t the du-paths.

5. Perform temporal testing automatically.

6. Examine the results

CEN 5076 Class 14 - 12/05 23

Program-Based Technique [Yang et al. 98]

Test coverage classification:

• acceptable – du-path in P free of infeasible paths

• unacceptable – violation of acceptable criteria

• w-runnable – du-path that does not cause an infinite wait in any thread (liveness or safety?)

• non-w-runnable – violation of w-runnable

CEN 5076 Class 14 - 12/05 24

Program-Based Technique [Yang et al. 98]

Finding du-paths

1. Create Parallel Program Flow Graph (PPFG).

2. Annotate PPFG 1. DFS employed to cover required nodes (SYN-seqs

nodes and thread creation node)

2. Use dominator trees (DT) and implied trees (IT) approach to cover optional nodes

3. After a path is found all nodes are annotated with a traversal control number (TRN)

3. Path generation actual path coverage is generated using the traversal control annotations