V1 | 2019-11-19

Software Testing TechDay – November 19, 2019

Virtual System Testing with VectorCAST/QA, vVIRTUALtarget, vTESTstudio, & CANoe

2

CANoe, vTESTstudio, VT System,

vVIRTUALtarget,VectorCAST/QA

System Testing

Change-Based Testing

Virtual System Testing

Develop automated system test cases on a virtual ECU without any hardware present, and continuously run your automated

system test cases. Use Change Base Testing to quickly run the impacted system tests on each code change.

3

VectorCAST

System Testing for Develpers

System Testing for QA Teams

vVIRTUALtarget

Concepts

Use Cases

Conclusion

Today’s Demo

Questions and Answers

Agenda

V1 | 2019-11-19

The VectorCAST Solution for System Testing

VectorCAST/QA

5

Where VectorCAST Fits In Your Development ProcessVectorCAST

6

Our Product StrategyVectorCAST

Be the Tool of Choice for Developers

Provide the Most Automated Tools Possible

Enable Continuous and Collaborative Testing

Focus on Embedded Systems

7

The VectorCAST Philosophy, Make Testing Fast, Easy to run, ContinuousVectorCAST

Weeks MinutesTime to Test

Manual AutomaticLevel of Automation

Per Release Per ChangeFrequency of Test

QA EveryoneWho Runs Tests

8

Test FlavorsVectorCAST

System TestingUnit Testing Integration Testing

Individual units or modules are tested. It involves testing of source code by developers.

Individual modules are grouped together and tested. The purpose is to determine that modules are working as expected once they are integrated.

Testing is performed on the whole system by checking whether the system or application meets the requirement specification document.

9

An integrated workflow for system test execution and coverage analysis Compatible with all test execution frameworks

Code coverage measures testing completeness Change Based Testing (CBT) reduces total test time Probe Points allow whitebox system testing

Coverage By Analysis (CBA) supports manual analysis Requirements Gateway allows test to Requirement Linking

System Testing with VectorCASTVectorCAST

Coding

Architecture

Requirements Specification

System

Design

System

Testing

Integration Testing

Unit

Testing

10

VectorCAST

System Testing for Develpers

System Testing for QA Teams

vVIRTUALtarget

Concepts

Use Cases

Conclusion

Today’s Demo

Questions and Answers

Agenda

11

System Testing VectorCAST/QASystem Testing for Develpers

Automated System TestingReduces Total Test Time

Source Code MetricsTest Pass/Fail

Code CoverageVectorCAST/QA

Test Infrastructure

BuildSystem

12

What developers are looking forSystem Testing for Develpers

Whitebox Testing without all the setup required for Unit Testing

Blackbox View for the big picture but ability to easily drill down to help with find bugs Integration with their current System Testing Tools Fast and Easy

13

Drill down with Probe PointsSystem Testing for Develpers

14

Blackbox View of the SystemSystem Testing for Develpers

15

VectorCAST and CANoe Complements Each OtherSystem Testing for Develpers

System Testing

CANoe and VectorCAST/QASystem Testing

CANoe

Unit Testing

VectorCAST/C++

ECU I/O (Bus, A/D) ECU I/O (Bus, A/D)

CANoe is used to perform testing of an ECU via the ECU’s external interfaces.

Test Driver

Adding VectorCAST/QA exposes the completeness of testing, providing a detailed look on the internal structure of the embedded software.

Adding VectorCAST/C++ allows testing of hard-to-reach sections of the application.

16

Integration of VectorCAST and CANoe/vTestStudio

System Testing for Develpers

Objective: Provide a detailed look on the internal structure of the embedded software during CANoe system tests

Instrument source code of SUT for code coverage with VectorCAST/QA

Automatically execute tests with VectorCAST/QA and CANoe, system test cases are created with vTESTstudio

Analyze code coverage with VectorCAST/QA

ECUSystem under Test

Functionality on the granularity of test runs already available

Tight integration to fully support test case oriented coverage planned

17

VectorCAST

System Testing for Develpers

System Testing for QA Teams

vVIRTUALtarget

Concepts

Use Cases

Conclusion

Today’s Demo

Questions and Answers

Agenda

18

What QA Teams are looking forSystem Testing for QA Teams

Blackbox View of the system

Test execution efficiency and optimization Certification Reports

Tool that supports their Software Development Process objectives (DevOps, CI, Quality Objectives) Simple code snippets (Probe Points)

19

VectorCAST 2019 TÜV SÜD CertificateSystem Testing for QA Teams

20

Blackbox View of the SystemSystem Testing for QA Teams

21

Comparing changes is key to assessing risk

Determine if a code change affects other parts of the system Prioritize tests based on risk, change, and criticality of modules

Change-based testing permits prioritized tests of modified modules Regression testing ensures changes do not introduce new faults

Change Based TestingSystem Testing for QA Teams

A.cpp

ATest1

ATest2

ATest3

ATest4

ATest5

ATest6

ATest7

ATest8

ATest9

ATest10

ATest11

ATest12

ATest13

ATest14

ATest15

B.cpp

BTest1

BTest2

BTest3

BTest4

BTest5

BTest6

BTest7

BTest8

BTest9

BTest10

BTest11

BTest12

BTest13

BTest14

BTest15

C.cpp

CTest1

CTest2

CTest3

CTest4

CTest5

CTest6

CTest7

CTest8

CTest9

CTest10

CTest11

CTest12

CTest13

CTest14

CTest15

CTest1

CTest2

CTest3

CTest4

CTest5

ATest6

ATest7

ATest8

ATest9

ATest10

BTest11

BTest12

BTest13

BTest14

BTest15

CTest11

CTest12

CTest13

CTest14

CTest15

22

Change Based Testing – “Test Less”, “Fail Faster”System Testing for QA Teams

Source

Code

Unit, Integration, System Tests

VectorCAST determines which test cases have been affected by a source change

Source Change

Test Cases to be re-run

23

Combine CoverageSystem Testing for QA Teams

100%

Coverage

Achieved

Functional Test/System Test

Unit Test

Driver Units Stubs

Code Coverage

60%-70%

Coverage

Achieved

30%-40%

Coverage

Achieved

24

Use VectorCAST/QA to automate your system tests

Generate coverage metrics to assess the quality of your tests Automate the generation of quality reports Use VectorCAST/QA with your existing System Tests

Leverage Probe Points to enable White Box system testing of your application Use Unit Testing to enhance the quality of your test coverage

SummarySystem Testing for QA Teams

V1 | 2019-11-19

The MICROSAR Solution for Virtual Integration

vVIRTUALtarget

26

Generic platform (PC) rather than real ECUs

Comfortable test environment and debugging Reusable tests across the development process Early start of integration and test without hardware

Smooth transition from virtual to hardware target

Motivation of VirtualizationvVIRTUALtarget

t

Virtual EnvironmentIntegration Platform

DevelopmentPhase

Interactive and Automated TestsTest

Start of Integration

A-Sample B-Sample C-Sample SOPPrototype

Real environment

27

Applications in the Development ProcessvVIRTUALtarget

Model-based Development

SWC IntegrationVFB Emulation

SWCA

SWCB

SWCC

Service Emulation

BSW

RTE

SWCA

SWCB

MCALOS

ECU Integration

System Integration

virtual virtual

virtual

VTT pro

VTT basic

vVIRTUALtarget(VTT)

28

vVIRTUALtarget pro Additional test focus:

> Testing of SWCs and SWC Compositions with VFB emulation Input artifact: AUTOSAR SWC description(s) Includes vVIRTUALtarget basic license & features

vVIRTUALtarget basic> Integration environment for ECUs with MICROSAR4 stack> Test focus:

> Validation of RTE and BSW configuration> Functional testing of the whole ECU software (application + BSW)

> Input artifact: AUTOSAR ECU extract

vVIRTUALtarget

29

VectorCAST

System Testing for Develpers

System Testing for QA Teams

vVIRTUALtarget

Concepts

Use Cases

Conclusion

Today’s Demo

Questions and Answers

Agenda

30

ArchitectureConcepts

Application

MICROSAR RTE

MICROSAR BSW

vVIRTUALtarget Emulation Library

vVIRTUALtarget Controller

CANoe

vVIRTUALtarget MCAL

vVIRTUALtarget OS

CPU

Interrupt Contr.

31

vVIRTUALtarget Modules in MICROSARConcepts

3rd Party SoftwareVector Standard Software 1 Includes ADCEXT, EEPEXT, FLSEXT, ETHDRVEXT and WDGEXT2 Full API compatibility, but no transmission functionality

Vector vVIRTUALtarget

OS

Application

RTE

OS

ComplexDriver

Microcontroller

SCHM

Application

HSMvHSM

E2E Protection Wrapper

SYS DIAG

CRYPTO

AMD

BSWM

COMM

DET

ECUM

STBM

TM

WGDIF

WGDM

CSM

CRYIF

CRYPTO(SW)

DCM

DEM

vDES

FIM

J1939DCM

vDRM

vDBG

DLT

vRTM

XCP

MEM

OTA

CAN

EA

FEE

MEMIF

NVM LIN FRJ1939TP

J1939NM

J1939RM

CANXCP

CANTP

CANNM

CANSM

CANTSYN

CANIF

vLINXCP

LINTP

LINNM

LINSM

LINIF

FRXCP

FRTP

FRARTP

FRNM

FRSM

FRTSYN

FRIF

COMCOM LDCOM IPDUM NM PDUR

COMXF SOMEIPXF E2EXF SECOC SOMEIPTP

ETHETHXCP

UDPNM

SD

DOIP

SOAD

vETM

vTLS

TCPIP

ETHSM

ETHTSYN

vETHFW

ETHIF

IO LIBS

V2G

AVB

V2X

vDIOHWAB

vSENT

CRC

E2E

vDNS

vEXI

vHTTP

vSCC

vXMLSecurity

vRTP

vAVTP

vSRP

vPTP

V2XBTP

V2XFAC

V2XGN

V2XM

MCALADCDRV

CANDRV

CORTST

DIODRV

EEPDRV

ETHDRV

ETHSWTDRV

FLSDRV

FLSTST

FRDRV

GPTDRV

ICUDRV

vIICDRV

LINDRV

MCUDRV

OCUDRV

PORTDRV

PWMDRV

RAMTST

CRYPTO(HW)

SPIDRV2

WDGDRV

WETH

CRYPTO(vHSM)

EXTCANTRCV

DRVEXT1

ETHTRCV

FRTRCV

LINTRCV

vSBC

vPSI5 DRV

WETHTRCV

vLhyp

vOtaDL

32

Virtual ECU is compiled and debugged with Microsoft Visual Studio

Runtime environment is CANoe Use CANoe as your test tool

Virtualization with vVIRTUALtarget basicConcepts

PC

BSW

RTE

SWCA

SWCB

SWCC

HW MCAL

HWOS

Development Environment

Conventional Setup

Debug Interf.

Bus Interf.

PCVirtual ECU

BSW

RTE

SWCA

SWCB

SWCC

VTT MCAL

VTTOS

Visual Studio

Setup with VTT

debug

Test Tool stimulateobserve Test Tool

33

Comfortable simulation environment with rich test feature set

Simulation can be run “as fast as possible” to shorten test cycles Virtual ECUs can be tested

with simulated bus with real bus and connected (real) ECUs, VT System, etc. with Functional Mockup Units (FMU)

CANoe as Runtime Environment and Test ToolConcepts

PCCANoe

BSW

RTE

SWCA

SWCB

SWCC

VTT MCAL

VTTOS

Visual Studio

Test featureset

Bus Interface

debug

34

One configuration for both VTT and hardware target Non-MCAL configuration is used for both VTT and hardware Virtual MCAL/OS configuration is synchronized with hardware MCAL/OS configuration

Generate either for the hardware target or for the virtual target depending on your development and test focus

Increased Development Efficiency Minimal effort for keeping virtual

and hardware configuration in sync Minimal effort to switch between virtual

and hardware target

Dual TargetConcepts

SWC B

MCAL

OS

Auto-sync of VTT modules

BSW

RTE

SWC B SWC B

35

One Configuration for Virtual and Hardware TargetConcepts

MICROSAR SIP

Virtual ECU Integration

ECU Integration

Generate Code

.HEX

.DLL

.c/.h Application source code

.c/.h

.c/.h

.c/.h

.c/.h

Dual-Target Configuration

SWC B

+ SIP Add-On VTT

DaVinci Configurator Pro

MCAL

OS

HW Module

VTT Module

Auto-sync of VTT modules

BSW

RTE

SWC B SWC B

36

Switching targets is easy!Concepts

Virtual (VTT)

Real (hardware)

‘One Source’ Development

Running and Testing on your PC with vVIRTUALtarget Hardware-independent features

Running and Testing on real target hardware Hardware-dependent features

Simply choose the target when generating your project No hassle with synchronization of separate AUTOSAR projects

Use the same (non-MCAL) BSW configuration to generate for virtual and hardware target

37

VectorCAST

System Testing for Develpers

System Testing for QA Teams

vVIRTUALtarget

Concepts

Use Cases

Conclusion

Today’s Demo

Questions and Answers

Agenda

38

Evaluation of the MICROSAR4 stack Evaluation Bundle VTT (VTT Only) Evaluation Bundle VC121 & VTT (Dual Target)

Prototyping and serial production projects VTT Beta SIP for early start (VTT Only) Beta/Prototype/Production SIP with SIP Add-On VTT (Dual Target)

Use Cases

Bundle/SIP Types:

VTT Only: Only Virtual Target is supported Dual Target: Both Virtual and Hardware Target is supported

39

MCAL: Hardware Specific

Compiler: Hardware Specific

Execution Platform: Microcontroller

VTT Beta SIPvVIRTUALtarget basic: VTT only

MCAL: vVIRTUALtarget basic

Compiler: MS Visual Studio

Execution Platform: CANoe/Executor

VTT Beta SIPUse Cases

Order

Beta SIP Production SIP

Scope of Integration and Test

- Integrate BSW with customer databases

- Integrate and develop application (e.g. SWCs, IoHwAb)

- Start functional testing e.g. using CANoe test features

Scope of Integration

- Integrate and configure hardware MCAL

- Complete functional testing

- Hardware specific tests

- Non-functional tests (e.g. runtime)

Project

Migration

Optional: SIP Add-On VTT

vVIRTUALtarget basic: testing in virtual environment

40

Distributed SWC Development with VTTUse Cases

Integrator (OEM/TIER1) SWC Supplier (TIER1/TIER2)

SWC A

SWC B

MCAL

OS

BSW

RTE

SWC A

SWC B

SWC Design

SWC A

VFB Emu

SWC implementation and testin the virtual environment

VTT pro VTT basic

VFB Emulation

SWC A

SWC B

VTT proSWC Description

Functional test on RTE/VFBlevel

Integration test in SWC composition

.obj.obj.obj

Functional test of SWCs Integration into MICROSAR

stack

.obj.obj.c/.h

.lib.lib .obj.obj.obj

Visual StudioCompiler

.lib

.arxml

Compiler forTarget Hardware

Static Libraryfor VTT

Object Code forHardware

2.

1.

41

Distributed ECU Development with VTTUse Cases

OEM ECU Supplier (TIER1)

ECUB

Network and ECU Design

SWC implementation and test ECU integration Validation of ECU configuration

ECU Extract

Execution of virtual ECUs supplied by the TIER1 Mixed test setup of virtual and real ECUs

.arxml

VTT basic

ECUA

ECUC

VTT MCAL

VTTOS

BSW

RTE

SWC SWC SWC

MCAL

OSBSW

RTE

SWC SWC SWC

Migration

VTT Only

VTT Dual Target

.dll

.dll

.hex

Target Binary

VirtualECU

A

.dllVirtual

ECUB

VirtualECU

C

Node Layer DLL,CAPL, .NET, …

VirtualECU

A

.dll

.hex

Virtual ECU

Virtual ECU

2.

1.

4.

3.

42

vVIRTUALtarget basic requires the following products: MICROSAR4 SIP with SIP Add-On VTT DaVinci Configurator Pro vVIRTUALtarget pro or basic license CANoe (minimum v8.5) Microsoft Visual Studio 2013 (Express or Ultimate) or 2017 (Community or Professional)

vVIRTUALtarget basic – Overview of products and optionsConclusion

43

vVIRTUALtarget basic ... … is a PC-based execution platform for MICROSAR4 ECUs … can be used to set up and run an ECU project before the hardware is available … dual projects enable seamless switching between hardware and virtual target

vVIRTUALtarget basic is integrated into CANoe Stimulate, test and debug the virtual ECU Make use of powerful CANoe test features Connect further hardware to test virtual ECUs in ECU networks

Key Features and AdvantagesConclusion

44

VectorCAST

System Testing for Develpers

System Testing for QA Teams

vVIRTUALtarget

Concepts

Use Cases

Conclusion

Today’s Demo

Questions and Answers

Agenda

45

Integration of VectorCAST and CANoeToday’s Demo

White Box System Testing Instrument source code of SUT for code coverage with VectorCAST/QA Automatically execute tests with CANoe, system test cases are created with vTESTstudio Analyze code coverage with VectorCAST/QA

ECUSystem under Test

46

Change-Based TestingToday’s Demo

Test CasesSource code traceability

Code Change

Source CodeInstrumented Source Code

Do we have to execute all test cases if the source code changes?

Without Change-Based Testing Running all tests take a long time

Many tests are complicated to run Test results are difficult to interpret

Do we have to execute all test cases if the source code changes?

Change-Based Testing Code coverage provides traceability between source

code and test cases: Which source code is covered by which test case

Do we have to execute all test cases if the source code changes?

Change-Based Testing Code coverage provides traceability between source

code and test cases: Which source code is covered by which test case

VectorCAST determines which test cases have been affected by a source code change

Only test what has changed to reduce test time, find errors faster and improve testing efficiency

47

Virtual System TestingQuestions and Answers

!?

48 © 2019. Vector North America Inc. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1 | 2019-11-19

Author:Kurt Krueger

More Information!Visit our Website for:> News> Products> Demo Software> Support> Workshops> Contact Addresses

www.vector.com