1

Dynamic Software Verification with GNAT Martyn PikeOctober 1st 2015

2

What will you learn today ?AdaCore are InventiveGNAT Technologies go beyond the Implementation Phase

AdaCore are CollaborativePartners and Third-party Technology for Dynamic Verification

AdaCore are OpenWork with Open Source, Open APIs and Open Standards

AdaCore are DependableCommitted to helping our customers address their challenges

Dynamic Verification ChallengesRun Cross Compiled Code on Target HardwareSolution : Target Architecture Emulation

Achieve a Planned Level of Code CoverageSolution : Coverage Analysis Tool

Managing the Verification EnvironmentSolution : Unit Test Harness Generator

Monitoring the Verification ProcessSolution : Monitoring Technical Debt

4

GNAT for Dynamic Verification

TARGET EMULATIONCOVERAGE ANALYSISUNIT TEST GENERATIONGNATemulatorGNATcoverageGNATtestMANAGEMENTGNATdashboard

5

with Ada.Text_IO;with Simple;

procedure Main is

Even_Count : Integer := 0; Odd_Count : Integer := 0;

begin

Simple.SubP( Loop_Count => 21, Even_Count => Even_Count, Odd_Count => Odd_Count );

Ada.Text_IO.Put_Line("Evens : " & Even_Count'Img); Ada.Text_IO.Put_Line("Odds : " & Odd_Count'Img);

end Main;package Simple is

procedure SubP( Loop_Count : in Integer; Even_Count : out Integer; Odd_Count : out Integer ) with Pre => (Loop_Count > 0), Post => ((Even_Count >= 0) and (Odd_Count >= 0));

end Simple;package body Simple is

procedure SubP( Loop_Count : in Integer; Even_Count : out Integer; Odd_Count : out Integer ) is begin

Even_Count := 0; Odd_Count := 0; For_Loop: for I in 1..Loop_Count loop if I > 20 then if I mod 2 = 0 then -- Unreachable when Loop_Count = 21 ! Even_Count := Integer'Succ(Even_Count); end if; elsif I mod 2 = 0 then Even_Count := Integer'Succ(Even_Count); else Odd_Count := Integer'Succ(Odd_Count); end if; end loop For_Loop;

end SubP;

end Simple;

6

GNATemulatorQEMUOpen Source Processor Emulator

I/O ConnectionsAda.Text_IOGNATbusGDB Connection

No InstrumentationActual Target Object Code

EXE

PowerPC -> x86 TranslationHost Platform

7

GNATemulator Platform SupportVxWorks 6VxWorks 653VxWorks 7 (coming soon)

PowerPC ELF Bareboard

LEON ELF Bareboard

ARM ELF Bareboard (coming soon)

8

GNATemulator Demonstration

9

Triage by Verification EngineerReproduce Defects

GNATemulator DebuggingEXE

GNATemulatorGDBSERVER

TCP

10

GNATemulator Debugging

11

GNATcoverageCoverage Analysis

Multiple Modes of Operation

Run with Capture of Execution Trace Data

Conversion of IEEE-ISTO 5001-2003 (Nexus) Trace Data

Coverage Analysis of Execution Trace DataSource Code LevelObject Code Level

12

GNATcoverage Platform SupportIntel 32-bit and 64-bit

PowerPC ELF Bareboard

LEON 2 & LEON 3 ELF Bareboard

13

Build ConsiderationsCompiler Command Line Switches-g-fpreserve-control-flowControl Optimizers for precise SLOC info-fdump-scosSource Coverage Obligation in ALI files

Support for Optimizations (upto -O1)Inlining Allowed (-gnatn)No External Libraries Needed

14

GNATcoverage Execution

15

GNATcoverage Analysis LevelsSource Levelgnatcov coverage --level=stmtgnatcov coverage --level=stmt+decisiongnatcov coverage --level=stmt+mcdc

Object Levelgnatcov coverage --level=insngnatcov coverage --level=branch

16

GNATcoverage Analysis FormatsxcovAnnotated Sources in Text Format

reportTextual Summary

HTMLColours, Sortable Columns and Per-project indexes

17

GNATcoverage Source Statementgnatcov coverage --level=stmt --annotate=dhtml

18

GNATcoverage Source Statement

19

GNATcoverage Source Decisiongnatcov coverage --level=stmt+decision --annotate=dhtml

20

GNATcoverage Source Decision

21

H:\tech_day_eu\code\src\simple.adb:80% of 10 lines coveredCoverage level: stmt+decision 1 .: package body Simple is 2 .: 3 .: procedure SubP( 4 .: Loop_Count : in Integer; 5 .: Even_Count : out Integer; 6 .: Odd_Count : out Integer 7 .: ) is 8 .: begin 9 .: 10 +: Even_Count := 0; 11 +: Odd_Count := 0; 12 .: 13 +: For_Loop: 14 +: for I in 1..Loop_Count loop 15 .: 16 +: if I > 20 then 17 .: 18 !: if I mod 2 = 0 then 19 .: 20 .: -- Unreachable when Loop_Count = 21 ! 21 -: Even_Count := Integer'Succ(Even_Count); 22 .: 23 .: end if; 24 .: 25 +: elsif I mod 2 = 0 then 26 .: 27 +: Even_Count := Integer'Succ(Even_Count); 28 .: 29 .: else 30 .: 31 +: Odd_Count := Integer'Succ(Odd_Count); 32 .: 33 .: end if; 34 .: 35 .: end loop For_Loop; 36 .: 37 .: end SubP; 38 .: 39 .: end Simple;

22

** COVERAGE REPORT **

============================= 1. ASSESSMENT CONTEXT =============================

Date and time of execution: 2015-09-15 14:36:12.00Tool version: XCOV 1.3.1 (20150118)

Command line:

H:\Tech_Day_EU\tools\gnatcov\bin\gnatcov.exe coverage --level=stmt --annotate=report --target=powerpc-elf -Pex3.gpr exe\ex3\ex3.trace

Coverage level: stmt

Trace files:

exe\ex3\ex3.trace program: exe\ex3\ex3 date : 2015-09-15 12:35:06 tag :

============================== 2. COVERAGE VIOLATIONS ==============================

2.1. STMT COVERAGE------------------

simple.adb:21:16: statement not executed

1 violation.

=========================== 3. ANALYSIS SUMMARY ===========================

1 STMT violation.

** END OF REPORT **

23

GNATcoverage Object Instructiongnatcov coverage --level=insn --annotate=dhtml

24

GNATcoverage Object Instruction

25

GNATcoverage Object Instruction

26

GNATcoverage Object Branchgnatcov coverage --level=branch --annotate=dhtml

27

GNATcoverage Object Branch

28

GNATtestUnit Test Harness Generation

Included in GNAT Pro

Based on the Open Source AUnit Framework

Available for Native and Cross Compiled Code

Increase Productivity Through Code Generation

29

GNATtest Work Flow

HARNESS

TEST CASESUUTpackage Simple is

procedure SubP( Loop_Count : in Integer; Even_Count : out Integer; Odd_Count : out Integer ) with Pre => (Loop_Count > 0), Post => ((Even_Count >= 0) and (Odd_Count >= 0));

end Simple;

30

GNATtest Work Flowgnattest -v --RTS=ravenscar-full-prep -Pex4.gpr

Generates Test Case Skeletons and a Harness

Very Flexible and allows for Configuration Control

Aware of User Defined Test Case Code

Clearly identifies what will NOT survive a Generation

31

GNATtest Test Case Generationpackage Simple.Test_Data is

-- begin read only type Test is new AUnit.Test_Fixtures.Test_Fixture-- end read only with null record;

procedure Set_Up (Gnattest_T : in out Test); procedure Tear_Down (Gnattest_T : in out Test);

end Simple.Test_Data;-- This package has been generated automatically by GNATtest.-- Do not edit any part of it, see GNATtest documentation for more details.

-- begin read onlywith Gnattest_Generated;

package Simple.Test_Data.Tests is

type Test is new GNATtest_Generated.GNATtest_Standard.Simple.Test_Data.Test with null record;

procedure Test_SubP_75ecda (Gnattest_T : in out Test); -- simple.ads:3:4:SubP

end Simple.Test_Data.Tests;-- end read only

32

GNATtest Test Case Generation-- This package is intended to set up and tear down the test environment.-- Once created by GNATtest, this package will never be overwritten-- automatically. Contents of this package can be modified in any way-- except for sections surrounded by a 'read only' marker.

package body Simple.Test_Data is

X : constant Integer := 20;

procedure Set_Up (Gnattest_T : in out Test) is pragma Unreferenced (Gnattest_T); begin null; end Set_Up;

procedure Tear_Down (Gnattest_T : in out Test) is pragma Unreferenced (Gnattest_T); begin null; end Tear_Down;

end Simple.Test_Data;

33

GNATtest Test Case Generation-- This package has been generated automatically by GNATtest.-- You are allowed to add your code to the bodies of test routines.-- Such changes will be kept during further regeneration of this file.-- All code placed outside of test routine bodies will be lost. The-- code intended to set up and tear down the test environment should be-- placed into Simple.Test_Data.

with AUnit.Assertions; use AUnit.Assertions;

package body Simple.Test_Data.Tests is-- begin read only procedure Test_SubP (Gnattest_T : in out Test); procedure Test_SubP_75ecda (Gnattest_T : in out Test) renames Test_SubP;-- id:2.2/75ecda11d3241da6/SubP/1/0/ procedure Test_SubP (Gnattest_T : in out Test) is -- simple.ads:3:4:SubP-- end read only

pragma Unreferenced (Gnattest_T);

begin

AUnit.Assertions.Assert (Gnattest_Generated.Default_Assert_Value, "Test not implemented.");

-- begin read only end Test_SubP;-- end read only

end Simple.Test_Data.Tests;

34

GNATtest Test Case Completion Even_Count, Odd_Count : Integer;

begin

Simple.SubP( Loop_Count => 21, Even_Count => Even_Count, Odd_Count => Odd_Count ); Assert(((Even_Count = 10) and (Odd_Count = 10)),"Loop_Count => 21"); Simple.SubP( Loop_Count => 22, Even_Count => Even_Count, Odd_Count => Odd_Count ); Assert(((Even_Count = 11) and (Odd_Count = 10)),"Loop_Count => 22");

35

GNATtest Harness GenerationHierarchy of GNAT project files

Integrates Test Case Code with AUnit Framework

Flexibility

Build an individual executable per test case

Build a single test case which runs all test cases

GNATtest Harness Execution

37

GNATtest Coverage Analysis

38

GNATtest Coverage Analysis

39

GNAT Programming Studio (GPS)

40

GNATdashboardMonitoring Dynamic Verification

GNAThub Data Aggregator

Open Python API

Roadmap [Q1 2016]Integration with GNATcoverageSupport for GNAT Pro Cross Compilers

41

GNAThub Data Aggregatorproject Hub5 is

for Object_Dir use "out\ex5\decision";

package Dashboard is for Plugins use ("GNATCoverage"); end Dashboard;

end Hub5;gnathub --exec my_script.py -P hub5.gprParses xcov Coverage Reports found in Object_DirBuilds SQLite snapshot database and executes my_script.py

42

Open Python APIGenerated SQLite Database can be accessed by Python script

GNAThub APIImplemented in Ada and exported to Python

Integrate the GNAThub Data into customer QA environment

Databases Delivered from External Teams or put under SCM

43

44

Partner SolutionsGNATcoverage Integration

On-Chip Analyser

Freescale MPC5634M SoC

IEEE-ISTO 5001-200 (Nexus)

45

Partner SolutionsVectorCAST/AdaAda 2012 support Q12016

VectorCAST/CoverJoint Webinar on CodePeer 3.0 Integration

Fully Integrated with GNATemulator supporting VxWorks

46

Partner SolutionsRapita Verification Suite (RVS)

On-target timing analysis (RapiTime)

RTOS-independent visualization of scheduling and event tracing (RapiTask)

On-target code coverage measurement (RapiCover)

Tools are actually built using Ada, including some Ada 2012

47

Integration with GNAThub Open Python API

Partner Solutions

48

What you have learnt today ?AdaCore are InventiveGNAT Technologies go beyond the Implementation Phase

AdaCore are CollaborativePartners and Third-party Technology

AdaCore are OpenOpen Source, Open APIs and Open Standards

AdaCore are DependableCommitted to helping our customers address their challenges

ResourcesGNATcoveragehttp://www.adacore.com/gnatcoverage/

GNATemulatorhttp://www.adacore.com/gnatemulator

GNATtesthttp://www.adacore.com/gnatpro/toolsuite/gnattest/

GNATdashboardhttp://www.adacore.com/gnatpro/toolsuite/GNATdashboard/

Microsoft Engineering ExcellenceMicrosoft Confidential50