AdaCore Paris Tech Day 2016: Elie Richa - Integrated Unit Testing for a Trusted Code Generator

Elie Richa

Gem: Integrated Unit Testing for a Trusted Code Generator

QGen Developer and QA Coordinator

Model-driven Development of Critical Software •  An Automatic Code Generator (ACG) produces the source code from a

model-based specification

•  To avoid re-verifying the generated code each time the ACG must be "Qualified" (i.e. Trusted) (or “Qualifiable”)

•  Qualification requires a rigorous and thorough verification of the ACG §  Extensive testing of the ACG with high exhaustiveness

•  Testing is a major cost in ACG qualification

ModelAutomatic Code

Generator !(ACG)

Source Code

2

DO-178C Tool Qualification Levels

Criteria1:Atoolwhoseoutputispartoftheairborneso5wareandthuscouldinsertanerror.Criteria2:Atoolthatautomatesverifica=onprocess(es)andthuscouldfailtodetectanerror,and

whoseoutputisusedtojus0fytheelimina0onorreduc0onof:•  Verifica=onprocess(es)otherthanthatautomatedbythetool,or•  Developmentprocess(es)thatcouldhaveanimpactontheairborneso5ware.

Criteria3:Atoolthat,withinthescopeofitsintendeduse,couldfailtodetectanerror.

So$wareLevels,failureis…

LevelA:Catastrophic

LevelB:Hazardous

LevelC:Major

LevelD:Minor

3

TQL-1 Qualification: Requirements-Based Testing Model for Tools

ToolOpera=onalRequirements(TORs)

ToolRequirements(TRs)

TestCases

TestProcedures

SourceCode

implementedby

allocatedonToolArchitecture

implementedby

implementedby

verifiedby

implementedbyimplementedby

4

QGen

Architecture of QGen

5

Importer Preprocessor Sequencer

CodeModel

Generator(CMG)

Op=mizer Expander Printer

Simulink®

Model

Source Code

internal model representation internal code representation

Unit Testing Input Output

Integration Testing

Input Output

User Language Ti−1 TiMi

Source Code Ti+1

Unit testing vs. Integration Testing

Integration testing is preferred to unit testing

Objective Achieve unit testing exhaustiveness using only integration tests

Unit Testing Integration Testing

Test Data Editor/Viewer Internal languages have no editors User language has a good editor

Test Data Complexity Intermediate languages are complex User language is simpler, has higher abstraction

Test Exhaustiveness Achievable thanks to isolation of units Hard to achieve with no visibility on the internals of the tool

Mi+1

6

Executable Unit Test Cases

•  Write unit test cases as executable queries §  A query specifying the required input : Test Requirement §  A query specifying the expected output : Test Oracle

•  Example: functionSqrt(Arg:Float)returnFloat;

7

User Language Ti−1 TiMi

Source Code Ti+1Mi+1

Unit Test Requirements

tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test Oracles

… …

Test Case ID Test Requirement Test Oracle

1 Arg=0.0 Sqrt'Result=0.0

2 Arg>0.0andthenArg<=1.0 Sqrt'Result>0.0andthenSqrt'Result<=1.0

3 Arg>1.0andthenArg<=100.0 Sqrt'Result>1.0andthenSqrt'Result<=10.0

Instrumentation for Test Case Evaluation and Logging

•  All instrumentation is marked as ghost code

•  Compiler produces ghost code only when assertions are enabled

Log which test requirements are satisfied by the input data

Evaluate the test oracles of the covered test cases

Subprogram Under Test

control flow

instrumentation

instrumentation

8

Unit Test Cases

Integration Tests tr0,0 tr0,1 tr0,2 to0,0 to0,1 to0,2 tr1,0 tr1,1 to1,0 to1,1 tr2,0 tr2,1 to2,0 to2,1

Test0 SAT – – PASS – – – SAT – PASS – SAT – PASS

Test1 – – SAT – – PASS – SAT – PASS – SAT – PASS

Test2 – – SAT – – FAIL SAT – PASS – – SAT – PASS

Test3 – – – – – – SAT SAT PASS PASS – SAT – FAIL

Test4 SAT – – PASS – – – SAT – PASS – SAT – PASS

Non-covered Unit Test Case

Non-covered Unit Test Case Unit Test Failure Unit Test Failure

Integrated Unit Testing (IUT) Integration Test Input Ti−1 TiMi

Source Code Ti+1Mi+1

Unit Test Requirements

tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test Oracles

… … Test Execution

9

Achieving coverage of Unit Test Cases

•  To achieve coverage, Integration Testing must be thorough

•  Example: A Simulink® block type with 4 parameters §  3 possible types each (i.e. Scalar, Vector, Matrix) ⇒ 34 = 81 configurations §  Taking into account primitive types (int8, int16 …) ⇒ ~600000 configurations!

§  But wait, only 14 out of 81 are valid configurations (~7000 out of ~600000) §  The behavior is generally similar for all configurations, with subtle differences

§  How can we do this accurately for 120+ Simulink® block types?

•  Use the BlockLibrary specification language §  Specify variability parameters: input/output ports, data parameters

§  Specify constraints defining valid combinations of parameter values and types §  Factorise common specification elements §  Auto-generate a large test set with a meaningful/reasonable coverage criteria of the

possible configurations

•  Achieve Unit Testing using only integration test data (Simulink® models)

10

Integrated Unit Testing Framework

•  Other features of the Integrated Unit Testing framework include §  Checking the completeness and disjointness of test cases §  Storing temporary data specific to a test case to improve expressiveness §  Selecting/excluding specific groups of test cases in the instrumentation §  Optimising the instrumentation by only evaluating previously non-covered test cases §  Specifying test cases in separate files than the tested source code

•  Let us know if you’d be interested in Integrated Unit Testing for your code §  We are considering making it a GNATtest feature

11

Unit Test Requirements

Unit Test Oracles

Integration Test Input Ti−1 TiMi

Source Code Ti+1Mi+1

Unit Test Requirements

tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test Oracles

Test Execution

… …

Conclusion: Integrated Unit Testing •  Conventional Unit Testing of multi-pass code generators is painful

§  Difficult to create inputs and check outputs of individual passes

•  Integration Testing is generally easier §  Can use normal Models as input; generated Source Code as output

•  Integrated Unit Testing: accomplish Unit Testing using Integration Testing §  Instrument the tested tool with ghost code

§  Monitor internal data during Integration Testing

§  Log evidence of the coverage and passing of Unit Test Cases

•  The approach is applicable to other software §  Let us know if Integrated Unit Testing is interesting in your context

12