DO Qualification Kit

Simulink® Test™ Test Cases and Procedures

R2015b, September 2015

How to Contact MathWorks

Latest news: www.mathworks.com

Sales and services: www.mathworks.com/sales_and_services

User community: www.mathworks.com/matlabcentral

Technical support: www.mathworks.com/support/contact_us

Phone: 508-647-7000

The MathWorks, Inc.

3 Apple Hill Drive

Natick, MA 01760-2098

DO Qualification Kit: Simulink® Test™ Test Cases and Procedures

© COPYRIGHT 2015 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under

the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written

consent from The MathWorks, Inc.

FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the

federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees

that this software or documentation qualifies as commercial computer software or commercial computer software documentation

as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and

conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,

reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or

other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.

If this License fails to meet the government’s needs or is inconsistent in any respect with federal procurement law, the

government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.

Trademarks

MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a

list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective

holders.

Patents

MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more

information.

Revision History

September 2015 New for DO Qualification Kit Version 3.0 (Applies to Release 2015b)

v

Contents

1 Introduction ...................................................................................................................................... 1-1 2 Test Cases and Procedures ............................................................................................................... 2-1 3 Traceability Between Operational Requirements and Test Cases and Procedures ........................... 3-1 4 Executing Test Cases and Procedures .............................................................................................. 4-1 5 Test Results ...................................................................................................................................... 5-1

vi

1 Introduction

This document comprises test procedures (Reference DO-330 Sections 10.3.3.c) for the

following capabilities of the Simulink® Test™ verification tool:

Comparing simulation results or executable object code test results against expected results

to determine pass/fail simulation or test status.

Generating reports that contain simulation or test results, including pass/fail simulation or

test status, and requirements tracing.

This document is intended for use in the DO-330 tool qualification process for TQL-5 tools.

See also the DO Qualification Kit User's Guide, R2015b.

1-2

2 Test Cases and Procedures

Simulink Test Manager test files (*. mldatx), containing positive and negative test cases for

baseline, simulation, and equivalence tests, exercise the qualifying Simulink Test capabilities.

The test files invoke simulation of test models (*.slx), evaluate the simulation results according

to the pass/fail criteria specified in the test files (using *.mat files with expected results for

baseline test cases), and generate test reports. The test files run in a batch mode, and output test

reports are compared with expected results using Report Generator setup file. The qualification

test report summarizes the outcome of the comparison. Figure 1 shows testing workflow.

Microsoft® Windows® platform with Microsoft® Word 2013 or higher supports automated

comparison of .docx report formats. See Simulink Test Tool Operational Requirements and

Executing Test Cases and Procedures in this document.

Figure 1: Test Workflow

2-2

Requirements-based test cases and procedures have been developed in:

Simulink Test Tool Operational Requirements

To access the tool operational requirements document, on the MATLAB® command line, type

qualkitdo to open the Artifacts Explorer. The document is in Simulink Test.

The aggregated test procedure, qualkitdo_sltest_rs.rpt, is a MATLAB Report

Generator file. Executing the file:

Runs the test sub-procedures in batch mode.

Evaluates the qualification test outcome.

Generates a summary qualification report.

The file is located in:

matlabroot/toolbox/qualkits/do/sltest/

The test cases, associated test sub-procedures, and expected results are provided as:

Simulink test models with embedded test harness *.slx

Simulink Test Manager test files *.mldatx

MAT-files containing test vectors for baseline tests *.mat

Pre-generated test reports in all supported formats *.zip, *.pdf and *.docx

The files are located in:

matlabroot/toolbox/qualkits/do/sltest/tests/

The expected results are located in:

matlabroot/toolbox/qualkits/do/sltest/expected_results/

3 Traceability Between Operational Requirements and Test Cases and Procedures

A list of the test files and associations between them is provided in the traceability matrix.

Bi-directional traceability between the Tool Operation Requirements and test cases and

procedures is provided in a traceability matrix:

qualkitdo_sltest_trace.xlsx

The traceability matrix is located in:

matlabroot/toolbox/qualkits/do/sltest/

3-2

4 Executing Test Cases and Procedures

In the scope of Simulink Test Tool qualification process, the applicant will:

Review the Tool Operational Requirements for applicability to the project under

consideration and select the relevant test procedures to be run in the installed user’s

environment (activity 6.2.1.aa of DO-330)

Configure and execute the test procedures in the installed user environment

Evaluate the generated test reports are correct (activities 6.2.1.b and 6.2.1.bb of DO-

330)

Configure shipped test data and generated test reports in a configuration management

system (activities 7.2.7 of the of DO-330)

Note:

To run the test cases and procedures, copy the folder

matlabroot/toolbox/qualkits/do/sltest and its subfolders to a location

where you have write access, and then run the test procedures from there.

Before you execute the test procedures, start a new MATLAB® session (the Report

Generator™ setup file runs MATLAB code in global workspace and might interfere

with data existing in the global workspace).

To execute the test procedure you must have the following licenses:

o MATLAB®

o Simulink®

o Simulink® Test™

o MATLAB® Report Generator™

o Simulink® Report Generator™

A Fixed-Point Designer™ license is required to qualify comparison of fixed-point data

types.

4-2

Execute the Simulink Test qualification test procedure:

1. Specify the Simulink Test report format to be qualified using the

qualkitdoCtrl.sltest.reportFormat variable in the base workspace:

At the MATLAB command line, enter

- qualkitdoCtrl.sltest.reportFormat = {'pdf'}; to generate

test reports in PDF format. PDF formats are not supported on Linux ®

platforms.

- qualkitdoCtrl.sltest.reportFormat = {'zip'}; to generate

test reports in HTML format compressed to zip archive.

- qualkitdoCtrl.sltest.reportFormat = {'docx'} to generate

test reports in Microsoft ® Word .docx format.

Note: If a report format is not specified, by default, the HTML format will be used.

Navigate to the folder where you copied the content of

matlabroot/toolbox/qualkits/do/sltest folder

2. Run the qualification test:

Using the GUI:

- Double-click the qualkitdo_sltest_rs.rpt file to open the Report

Explorer

- Select File > Report. Simulink Report Generator creates the summary report

and the model level reports

At the MATLAB command line, enter

report ('qualkitdo_sltest_rs.rpt')

Simulink Report Generator sequentially runs the Simulink Test Manager test files and

stores the generated Simulink Test reports in:

sltest/outputs/

4-3

After running the tests, Simulink Report Generator compares the generated Simulink

Test reports to the expected results (pre-generated reports that ship with the DO

Qualification Kit) and generates the summary qualification report. The summary

qualification report contains information about test environment, executed test files, and

qualification results.

Note: Automated comparison is supported for HTML and Microsoft® Word .docx

report formats, depending on the operational environment. If automated comparison is

not supported for selected formats, manually compare the reports to the expected

results. Automated report evaluation is supported in the following environments:

Platform Report Format

Linux ® HTML

Windows ® (32-bit and 64-bit) HTML, Microsoft ® Word .docx

3. After Simulink Report Generator has finished running the aggregate test procedure, verify

that there are no warning or error messages in the Simulink Report Generator message list

window. Review the summary qualification report. For more information, see Test Results.

If you do not have a Fixed-Point Designer license, the test file tFP.mldatx, containing

test cases for fixed-point data types, will not run. If you do not need to qualify comparison

of fixed-point data types, you can ignore this warning message.

4. In your environment, manually verify the requirement SLT_TIR_010:

a. Using Simulink Test Manager GUI, create a baseline test case for the Simulink

model mTIR010.slx, stored in the sltest/outputs/ folder.

b. Using the Simulink Test Manager GUI, capture the outputs of the model and store

it as a MAT-file.

c. Set the captured MAT-file as a baseline criteria with 0 absolute and relative

tolerances.

d. Run created test case. Ensure that it passes.

Note: Simulink Test does not provide an API to capture the simulation outputs.

This capability must be tested manually.

5 Test Results

Executing the aggregate test procedure file qualkitdo_sltest_rs.rpt generates and

stores the following data in /sltest/outputs/:

Summary qualification report qualkitdo_simulinktest_qualificationreport.html

Simulink Test reports in the specified format. The list of reports will be included in

summary qualification report.

The applicant will:

1. Review the test results:

If automated comparison of the reports is supported in your environment, review the

generated summary qualification report

qualkitdo_simulinktest_qualificationreport.html stored in the

sltest/outputs/ folder:

- Check that the information regarding test environment is correct and the list of

generated test reports includes all selected formats.

- In the “Qualification Test Results” section of the report, check that the

qualification status is evaluated as PASS.

Note 1: If the qualification status is evaluated as FAIL, manually review the

content of the differing test reports to assess the impact of the mismatch with

respect to the software life cycle activities performed using the tool.

Note 2: Automation comparison capability is verified in your environment using

the pre-generated reports shipped with the DO Qualification Kit. A list of

corresponding test cases is provided in the CMP page of the traceability matrix qualkitdo_sltest_trace.xlsx

5-2

If automatic comparison is not supported in your environment:

- Review the summary qualification report

qualkitdo_simulinktest_qualificationreport.html in the

sltest/outputs/ folder. Check that information regarding test environment

is correct and the list of generated test reports includes all selected formats.

- Compare each Simulink Test report file, listed in the section “Qualification Test

Results” of the summary qualification report, to the pre-generated version of the

report with the same file name provided in sltest/expected_results/.

Note: When comparing the reports dates, time stamps and test duration time may

differ between actual and expected results.

If you are not qualifying comparison of fixed-point data types, the summary report will

include a relevant note in the “Qualification Test Outcome” section.

2. Configure the content of the folder comprising the test cases, procedures and results in a

configuration management system.