Quality Assurance and Testing in LCG CHEP 2004 Interlaken, Switzerland 30 September 2004 Manuel Gallas, Jakub MOSCICKI CERN [email protected]

Quality Assurance and Testing in LCG

CHEP 2004 Interlaken, Switzerland 30 September 2004

Manuel Gallas, Jakub MOSCICKI

CERN

http://[email protected]

[email protected]

M.Gallas, J.Moscicki CERN PH-SFT

SPI - Software Process & Infrastructure 2

Outline:


•LCG SPI Context

•SPI Services Overview

•Sw-Testing:•Test Frameworks •Test Documents•User support

•Quality Assurance:•QA activities•Bug tracker statistics•Test coverage

•LCG App.Area policies

•Summary



LCG SPI context

LCG Infrastructure•Common services

•Similar ways of working (process)

•Tools, templates, training

•General QA, tests, integration, release

LCG Application Area software projects

•POOL: Persistency

•SEAL: Core common software

•PI: Physics Interfaces

•SIMU: Simulation

• …etc…

LHC grid software applications

(LHC experiments, projects, etc)

LCG SPI project

LCG Application Area

SIM

ULA

TIO

N



SPI Services Overview

Provide General Services needed by each project• CVS repository, Web Site, Software Library• Mailing Lists, Bug Reports, Task Management, Collaborative Facilities

Provide solutions specific to the Software Development phases• Tools, Templates, Policies, Support, Documentation, Examples• Software Testing and QA

Coding Analysis and Design Development

Release

SpecificationsTesting

Build systemsDeployment and

Installation Documentation

Quality Assurance

Software Development

General ServicesCVS service

Collaborative Facilities Task Management Mailing Lists

Web Portal External Software

SP

I p

roje

ct

The rest of the talk describes QA and Sw-Testing services

SPI Web Site - http://spi.cern.ch



•Software testing should be an integral part of the software development in the LCG App Area.•All level of software testing should be run as part of an automatic process.

SW-Testing Overview - http://spi.cern.ch/testing

Software testing

Integration test

System test

Acceptance test

Unit test

Sw-testing team

IntegrationTests

System Tests

ExamplesLHC experiments

Use in Exp.Test FrameWorks

SW-testing doc

SW- testing support

SPI SW-Testing

SW-testing polices

Code Documentation

Software developer

CODE

Work PackageTest

Automated testing

Nightly Building

System



LCG-SPI SW-Testing

Aim: to help developers: to help projects:

• to produce test code suitable to be run in automatic way • to document their tests

• to plan their test phase• to run all their test in automatic way • to browse and scan easily the test results and failures

Our inputs: • Contacts within HEP-community.• What is available as free open source code.

Our constraints: • Avoid commercial software and licensing problems. • Avoid “do it yourself solutions”•Try to adopt commonly used open-source software.

Requirements from the users : • Work with different languages (C++, Python, …) and in different environments (platforms & compilers)• Allow two ways of testing: (1) “check the output of the test-code” (2) “check the test results inside the test-code”.• Provide a easy way to integrate existing tests.• Allow the creation of dependencies among tests.• Easily establish a criteria if a test should pass or fail in a given release• Organize the tests by components or packages / integration / system /acceptance .• Regression testing and check respect to the previous sw-release.• Easy integration with the Nightly Building System.• Provide also a graphical interface to run the tests and examine the test results.



Test Frameworks: Global picture

X-Unit familyJUnit QtUnit

CppUnit PyUnit

Oval

Test Scripts

Old tests

SW Product Examples

Unit testing

Acceptanc

etesting

QA activity

End User Installation

checks

Nightly Building

Project Release

Sw-Testing

QMTest• Uses a GUI for creating and running tests (also in batch).• Can run tests in parallel, supports execution of a single test or many at once (test-cases & test-suites) • Organizes tests hierarchically• Records dependencies among tests

Top layer

Bottom layer

- Adaptable to theprogramming language and developer- Prepared to be run in automatic way

- Integrates different ways to test.

- Common environment to run the tests and to access the test results.



•Main Scope: Unit–testing

•The same “assertion style” in different languages.

•Provides:- The name of the test case that failed.

-The name of the source file that contains the test.

-The line number where the failure occurred

• Different platforms/compilers: (Linux/Solaris/Windows)

Test Frameworks: X-Unit family

X-Unit family

JUnit

QtUnit

CppUnit

PyUnit

A simple test:

1. Subclass the TestCase CppUnit or PyUnit class

2. Override the method runTest().

3. When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds



Test Frameworks: Oval

Oval:• validation and regression • developed & used within CMS experiment

•Can be use for Unit-test to Validation tests.

•Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”)

• It is possible to set different run environments.

• Can run external scripts and external binaries.

Ovalfile(configuration)

ExecutableTest

Source Code

Referencefile

Log file

Oval run

Ovaldiff

Oval build



Test Frameworks: QMTest

QMTest

•Uses a graphical interface for creating and running tests

•The configuration files are in XML and can be created from the GUI. We provide also script to do it

• Runs tests in parallel

• Organizes tests hierarchically

• Supports execution of a single test or many at once

• Records dependencies among tests

• Can be run in batch mode -> easy integration with the Nightly-Building systems



Test Documents

• For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities.• We use a template•Test Case template collects information about the existing tests and how they cover the expected functionality of the software.



User support: Web information

http://spi.cern.ch/

Test Frameworks

HowTo

Test doc

Policies

http://spi.cern.ch/testing



User support: How-To

*

*

3 Follow sw-testing policies

Automatic testing requires some standardization

1

2



Quality Assurance Service - http://spi.cern.ch/qa

• The main goal of QA activity is help LCG projects

• assess and improve the quality of the software

• provide tools to collect useful metrics/statistics which help toasses quality;

• generate reports;• verify if project setup is correct with LCG

policies.

• QA Tools and Focus • Automatic reports

• Development/integration of automatic tools

• LCG Policies• agreed and defined by AF• SPI supports them in the tools and

procedures and only helps to work them out



Quality Assurance Activities

QA Checklist on each Release• Build the release • Run automatic tests• Metrics

• Test Inventory & Metrics• Documentation/Examples Inventory • Bug tracker Statistics • Code Inventory • Rule Checker , Logiscope

• LCG Policies • Configuration of a build system • CVS directory structure • Software testing polices

Automatic QA Reports

• Generated at every release

• Published on the SPI web site

• External tools• Test coverage

• SLOC, Valgrind, ignominy

• QA procedure: well-defined, transparent, open- clear rules and checklist of assessed items- anybody at anytime may see statistics - anybody may create reports themselves- anybody may contribute



QA Reports

Bug tracker statisticsTesting inventory by packageResults of the automatic testingTest coverageSource code statistic (n files, n lines, sloc, .. )CVS structure



QA reports: Bugtracker Statistics

• Defect Analysis in Time Defect Reparation Trends Activity of the project / users Flexible category selection

subsystems, severity,... Caveat:

- not all bug reports are“real” bugs

• Correlation of defects and testsTraceability

oEvery bug fix should be accompanied with a test case

oTest case should contain a bug reference #Avoiding reappearing bugs



QA reports: Test Coverage

• Coverage of• Unit tests

• Integration tests

• Validation tests

• At the level of• Project

• Package

• File

• Source Lines

• Tools• gcc3.2 –gcov

• LTP scripts



LCG Policies

• LCG Policies• CVS and Build Directory Policy • Software Testing Policies • Version Numbers, Tagging and Release

Procedure • Installation Directory Structure • Platform string, binary names,

debug flags and more

• They are a needed by the LCG• They are defined by the LCG projects (AF),

collected by SPI• SPI supports them in the tools and

procedures and only helps to work them out

• If everything is different is too difficult to use and to automate

• compromising on our habits, for project needs

• tell when they are not followed

• First time that this exists at this extent, and that is checked

QA tools verify the compliance of software to the Policies



• QA activities• designed to match the HEP environment

- focused more on testing and bug tracking- less on sophisticated source code metrics

• integrated and automated into the LCG development environment

• QA infrastructure will be reused in EGEE

Summary


•LCG SPI Context

•SPI Services Overview

•Sw-Testing:•Test Frameworks •Test Documents•User support

•Quality Assurance:•QA activities•Bug tracker statistics•Test coverage

•LCG App.Area policies

•Summary

• Software testing activities• a complete and coherent automatic testing

infrastructure was deployed based on existing tools in the open source community. It covers covers from unit to validation tests.

• a unique entry point (QMTest) can be use to run all the tests at the release time, in the nightly building system, for the QA reports or by the developer.