EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

03/06/10Mark Heron

An Update on Automatic Regression Testing at Diamond

EPICS


03/06/10Mark Heron


EPICSContents

1. Motivation2. Update on the Testing Framework at

Diamond3. Hardware Regression Test4. EPICS Base Regression Test

03/06/10Mark Heron


EPICSIntroduction• Diamond is now operating for ~4500 hours per year

which will increase to ~5500 hours per year over the next two years

• Shutdown time is being reduced, and so are becoming increasingly busy– 2010 and 2011 see installation “Mini Beta Optics” in two

long straight sections necessitate replacing pairs of SR girders

• All this results in less time to test new software releases

• Drive to improve operational reliability– The Control System has not had significant number of

faults– Don’t want to change this

03/06/10Mark Heron


EPICSThe Grand Objective• Test all 200 support modules

– Test modules and EPICS DB templates

• Test device/drivers that interface to hardware– Hardware test environment

• Test the versions of EPICS base we are running – Issues of changing EPICS base and /Tool chain/OS/

version

• Test software against a tool chain, build and run environment – We want to ensure that operational system use the same

environment

03/06/10Mark Heron


EPICSGood Software Engineering Practice

• Four major parts to software engineering– Design: Should come first!

– Code: We are good at this.

– Documentation: Doxygen etc.

– Testing: Often manual and hard to repeat.

• Testing is not an optional extra– Can be carried out by the Software Developer

– Or can be carried out by the Software User

– Or can be automated

• Tests must be properly repeated – At least before every release and preferably more often

• Tests are easier to repeat if they are completely automatic

03/06/10Mark Heron


EPICSTest Framework• A software framework that aids the creation of

automatic test suites for modules.• Based on PyUnit, the Python unit testing library.• Test reports conform to the TAP protocol and/or

JUnit XML format.• Monitoring of EPICS database records for coverage

reporting.• Hardware and simulation targets can use same test

suite.

03/06/10Mark Heron


EPICSThe Run Script• Searches a directory tree for modules with automatic test

support.• Runs tests suites it finds, logging results and output.• Can run multiple tests in parallel subject to resource

constraints.• Level of diagnostic output can be specified.• Tests run can be restricted to single module, single target,

single case.• “dls-run-tests -h” for command help.• The run script is designed to operate from the ‘support’

directory of the work tree.

03/06/10Mark Heron


EPICSDirectory Structure Changes

etc

tests.py

<moduleName>

test

support

makeDocumentation

makeIocs

iocs

new directories

03/06/10Mark Heron


EPICSDevice Simulations• Based on existing DLS practice.

• Written in Python.

• Currently capable of simulating any device that communicates through serial or IP connection.

• Support for instrumentation to allow protocol coverage reporting.

03/06/10Mark Heron


EPICSFramework Class Diagram

03/06/10Mark Heron


EPICSExample – The Test Suitefrom pkg_resources import requirerequire('dls.autotestframework')from dls.autotestframework import *class Fw102TestSuite(TestSuite): def createTests(self): # Define the targets for this test suite Target("simulation", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex_sim.boot", epicsDbFiles="db/fw102Ex.db", simDevices=[SimDevice("controller1", 9016)], guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) Target("hardware", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex.boot", epicsDbFiles="db/fw102Ex.db", guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) # The tests CaseLocalIncrementSwitch(self) CaseLocalDecrementSwitch(self) CasePowerOffOn(self) CasePvIncrement(self) CasePvDecrement(self) CasePvMultipleIncrement(self) CasePvMultipleDecrement(self) CasePvSetPosition(self) CasePvTriggerMode(self) CasePvPolling(self)

03/06/10Mark Heron


EPICSExample – Intermediate Classclass Fw102Case(TestCase):

def curDevicePos(self): ''' Get the current wheel position from the device simulation ''' result = 0 self.command("controller1", "getpos") args = self.recvResponse("controller1", "pos", 1) if args is not None: result = int(args[0]) return result

def verifyPosition(self, intended): ''' Verify that the wheel is in the intended position ''' if self.simulationDevicePresent("controller1"): self.verify(self.curDevicePos(), intended) self.verifyPv("FGZ73762:WHEEL1:POSITION_RBV", intended) self.verifyPv("FGZ73762:WHEEL1:POSITION", intended) self.verifyPv("FGZ73762:WHEEL1:INPOS", 1)

03/06/10Mark Heron


EPICSExample – A Test Caseclass CaseLocalIncrementSwitch(Fw102Case): def runTest(self): '''The local increment switch.''' if self.simulationDevicePresent("controller1"): # Check the current position of the wheel before = self.initialPosition() # Take the wheel round twice for i in range(12): # Now advance the wheel using the local button self.command("controller1", "incr") after = before + 1 if after > 6: after = 1 # Check the final position of the wheel self.sleep(2) self.diagnostic("Before=%d, after=%d" % (before, after), 1) self.verifyPosition(after) before = after

03/06/10Mark Heron


EPICSExample - Test Report[1] 1..10 [fgz73762@pc0054 diamondtest]$ ./runtests.py -t simulation -f default.config -i -g -b -p 3 -l tests.log -q -m fw102

[1] ok 1 - The local increment switch.

[1] ok 2 - The local decrement switch.

[1] ok 3 - Power off and on.

[1] ok 4 - The PV increment command.

[1] ok 5 - The PV decrement command.

[1] ok 6 - Fast increment sync correction.

[1] ok 7 - Fast decrement sync correction.

[1] ok 8 - Movement directly to a position.

[1] ok 9 - Control of the trigger mode.

[1] ok 10 - Control of the polling mechanism.

[1] # ==============================

[1] # Passed 10/10 tests, 100.00% okay, in 260.57s

[1] #

[1] # ==============================

[1] # Sim device controller1 coverage report:

[1] # setpos: ok

[1] # settrigger: ok

[1] # getpos: ok

[1] # gettrigger: ok

[1] #

[1] # ==============================

[1] # EPICS database coverage report:

[1] # FGZ73762:WHEEL1:TRIGGER(mbbo): ok

[1] # FGZ73762:WHEEL1:TRIGGER_RBV(mbbi): ok

[1] # FGZ73762:WHEEL1:RESTART(ai): not touched

[1] # FGZ73762:WHEEL1:INPOS(calc): ok

[1] # FGZ73762:WHEEL1:REINIT1(fanout): ok

[1] # FGZ73762:WHEEL1:POSITION(mbbo): ok

[1] # FGZ73762:WHEEL1:COMMSFAIL(bi): ok

[1] # FGZ73762:WHEEL1:POSITION_RBV(longin): ok

[1] # FGZ73762:WHEEL1:CALCCOMMS(calcout): ok

[1] # FGZ73762:WHEEL1:DISABLEPOLL(bo): ok

[1] # FGZ73762:WHEEL1:STEPBK(calcout): ok

03/06/10Mark Heron


EPICSHudson• Use Hudson framework

– For scheduling and reporting of test

• A Continuous Integration Engine– Checkout from repository – Does some actions

• Build, Post-build actions, (execute test script)

– Determine Status

• Invoked by period checks on SVN repository• Status by web and emails to developers

• Success (all actions successful)• Unstable (some tests failed)• Failed (compile error)

03/06/10Mark Heron


EPICS

03/06/10Mark Heron


EPICSHardware Regression Test• Test IOC consists of:

– Prodex OMS-58 8 axis servo control card• A voltage to quadrature encoder conversion card

– Hytec 8001 DIO and 8515 Octal Serial– EPICS IOC test specific db

• Stimulation IOC:– 8005 – digital I/0 for limit switches, home switches– 8401 – ADC input card to monitor OMS-58 servo demand

output– 8402 – DAC to drive the encoder simulation– VxWorks application implemented by cut down drivers for

above cards and a simulation of some ‘realish’ hardware to control

• Test harness:– Wiring to join Test IOC to Stimulation IOC.

03/06/10Mark Heron


EPICSHardware Test Framework

Test Unit (EPICS IOC)

Hardware under test

Switch

Terminal Server

Stimulation Unit

(vxWorks stimulations)

Linux control PC

Test suite control,

redirection, etc

Power strip (IP)

Hudson

Processor Processor

Serial

Digital & Analog I/O

Test harness

Network

03/06/10Mark Heron


EPICSEPICS Base Regression Test Suite (1)• Came out of the Codeathon at Diamond

– Nick Rees, Jon Thompson and Matt Pearson

• Objective – To generate automated testing of the EPICS Base project

code on a diverse range of architectures, both workstation and real-time OSs.

– These will be run nightly at Diamond on head of trunk and results published on web site.

– Provide as service to the collaboration

• This is work in progress. Only started it last week.

03/06/10Mark Heron


EPICSEPICS Base Regression Test Suite (2)• There are established EPICS tests, but these are

run manually as part of releases• Wrap the established EPICS tests in Python to

make callable from test framework. • Run test on

– EPICS Base, C Unit tests– Report on test coverage for each EPICS Base test– IOCs based tests

03/06/10Mark Heron


EPICSEPICS Base Types of tests (1)• C unit tests based on epicUnitTest.h functions.

– These are mostly used to test libCom functions at this point

– Generate TAP output that is interpreted by the test harness

– Propose to add new make targets to describe how the tests are run on embedded architectures (give a download binary and a command line function to run)

– Make generates a test specification that is used by the test framework to run the tests

03/06/10Mark Heron


EPICSEPICS Base Types of tests (2)• IOC based system tests based on Diamond module

test framework– Python framework that loads IOC under test and a

stimulation system (if required)– Can get, set and monitor PV’s, and/or control or monitor

the IOC command line– Tests reside in a new etc/test directory in the application

tree– Have a wide range of verification commands. – Many of the EPICS “soft-test” suite tests were converted

to this framework during the Codeathon

03/06/10Mark Heron


EPICSEPICS Base Platforms

• Plan is to run this for popular development platforms and target architectures.

• What about historic development platforms, Solaris, and latest platforms say Fedora XX or Ubuntu XX.YY and other target architectures ?

Linu

x R

HE

L 4

/ G

CC

XX

X

Liln

ux R

HE

L5 /

G

CC

4.1

.2

Win

dow

s X

P/

MS

VC

Win

dow

s C

ygw

in

GC

C ?

??

VxW

orks

5.6

GC

C

3.4.

6

RT

EM

S ?

?? G

CC

??

?

Mac

OS

X G

CC

???

Linu

x G

CC

4.3

.2

Glib

c 2.

7

Linu

x G

CC

4.3

.2

Glib

c 2.

6.1

68030 MVME 167 x xPPC 7457 MVME5500 x x xx86 x x x x xARM xXilinx Virtex 2 PPC 405 x

Target OS / Toolchain and Library

03/06/10Mark Heron


EPICSEPICS Base Test Results• Results will be published as a web page containing:

– Results of ‘make runtests’ on the target architectures– Results of running the ‘soft-tests’ on the target

architectures– Code coverage report for the Linux host.

• URL to be announced

03/06/10Mark Heron


EPICSEPICS Base Test Results Web Interface

03/06/10Mark Heron


EPICSEPICS Base Test Results Web Interface

03/06/10Mark Heron


EPICS EPICS Base Test Results Web Interface

03/06/10Mark Heron


EPICSAcknowledgement

• This is being lead by Jon Thomson at Diamond

• It’s the work of many of the Controls Group at Diamond

Documents

EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond