Test suites for gLite CLIWork report

CERN-INTAS SA3 meetingJuly 4, 2008

Dubna, Russia

Dmitry Zaborov (ITEP-Moscow), Protvino group

An efficient way of testing command line interface Test suite for Data Management CLI (lcg_utils) Test suite for gLite WMS CLI Update on UI test suite

Outlook

• UI test suite has been updated– Now matches gLite 3.1

• Two new test suites have been developed– Both requested by [CERN] gLite certification group– Both can be viewed as specialized extensions of the UI test suite– Both are aimed to provide a maximally full coverage of possible

use cases and supplementary options of the CLI– Each covers an important and heavily used subset of gLite UI

commands– The tests are technically implemented in bash

• The test suites have common application domains– Middleware certification (currently main application)– Validation of new UI installations (can be recommended for

afs UI)– Bug testing

Efficient approach to CLI testingLCG DM and gLite WMS CLI represent sophisticated test targets• The number of options (n) and variants (m) of most commands is too large to

test every imaginable option combination one-by-one as this would result in m * 2^n ~ O(1000) test cases, making the test suite practically unusable.

• Fortunately, not every option combination is used equally often. Moreover, many options “run” completely independently and therefore do not need to be tested together. Thus one can drop most of the test cases from consideration and keep only the most used and representative ones, without substantial impact on the test suite coverage (=aggregation of detectable bugs and problems).

Several ideas are introduced to make the effort feasible• “Focus on one thing”: Look only for UI failures. Service-related and resource-

related failures should be tolerated whenever possible, i.e. when the failure is unlikely to be caused by a UI problem.

• “Disregard command history”: The order in which the commands are run is disregarded. The test cases are grouped as it is convenient. We believe such an approach is well justified for our client-oriented test suites because client libraries do not have memory to remember a erroneous state. We should note however that Grid services do have memory by definition, and therefore command ordering should not be neglected in service-oriented test suites.

• “Be simple”. No complex dependencies are assumed between different options. Only simple types of conflicts, such as “two options do not work together” are looked for.

Data Management test suite• Package org.glite.testsuites.ctb/DM• Tests are organized in 9 bash scripts. The only mandatory argument is SE

hostname (two SE hostnames for file replication test)– Example: DM-lcg-cr.sh lxb1921.cern.ch

• The following commands are tested: – lcg-cr, lcg-cp, lcg-del, lcg-la, lcg-lr, lcg-lg, lcg-aa, lcg-ra, lcg-rep lcg-rf, lcg-uf, lcg-gt,

lcg-sd, lcg-getturls, and lcg-ls

• Most of lcg- commands accept the name of source and/or destination files in different formats (SURL, LFN, GUID, gsiftp:, file:). All possible variants of these are covered by the test suite

– Example: DM-lcg-alias will test the following use cases of lcg-aa: lcg-la GUID, lcg-la SURL, lcg-la LFN, lcg-la ALIAS (second LFN)

• By default all the commands are tested without any special options. Additional options are supported by the test scripts which activate corresponding options of the lcg- commands. Options can be combined. In this way all important options of lcg- commands can be tested

– Example: ‘DM-lcg-ls.sh -v -t 20 --vo cms --nobdii --defaultsetype srmv2 lxb1921.cern.ch’ will test lcg-cr with all the most important options it accepts

• For more detail read the documentation: https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#Data_Management_DM

WMS CLI test suite• Package org.glite.testsuites.ctb/WMS/WMS-cli• Tests are organized in 16 bash scripts. No arguments or options should/need

to be supplied• The following commands are tested:

– glite-wms-job-delegate-proxy, glite-wms-job-list-match, glite-wms-job-submit, glite-wms-job-cancel, glite-wms-job-output, glite-wms-job-perusal and glite-wms-job-info, glite-wms-job-logging-info and glite-wms-job-status (last two are symbolic links to glite-job-logging-info and glite-job-status respectively)

• All more or less important options are tested• Two different approaches are combined in the test suite to achieve efficient an

structure: – “One option - one test“. This is used for the "basic cycle" tests, which include glite-

wms-job-submit, glite-wms-job-status, and glite-wms-job-output calls. The choice is dictated by the fact that job output can only be retrieved once for one job, and therefore one needs a job-submit for each job-output.

– “One command - one test". This approach is used for specialized and less used commands, such as glite-wms-job-list-match or glite-wms-job-perusal, and for certain options of glite-wms-job-submit.

• For more detail see https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#WMS_CLI

UI test suite updates

• Updates to match gLite 3.1– Remove obsolete commands (such as glite-job-submit) from the

test list– Account for a change in the behavior of some commands (e.g.

voms-proxy-init with -voms/confile/userconf options) and some Glue schema changes (certain attributes moved)

– Various minor improvements and fixes

• One new test– UI-myproxy-change-pass-phrase

• Twiki page is kept up-to-date– https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#User_Interface_UI

Practical results• The test suites are used for gLite middleware certification, thus

helping to prevent new bugs in gLite• Several bugs found

– Bug 35097: Segmentation fault in lcg-sd– Bug 34950: lcg-sd doesn't doesn't work with SRMv2 request token (already

fixed)– Bug 38073: Options -g and -l of lcg-rf do not work together– Bug 35698: Options --from and --to of glite-job-logging-info have one hour

offset (already fixed) – Bug 35706: Documentation refers to --config-vo option which is not

supported by the glite-wms commands• Certain known bugs are detectable by the test suites (so can be

monitored)– Bug 24963: glite-job-submit --valid hh:mm fails– Bug 34420: WMS Client: glite-wms-job-submit option --valid does not

accept any time value– Bug 30989: glite-wms-job-status -all systematically fails– Bug 36573: voms-proxy-info has bad return codes– Bug 33459: voms-proxy-info on non-VOMS proxy gives a misleading error

Conclusion

• Two new test suites have been developed and cover two important sets of gLite commands– Data Management CLI (lcg_utils)– gLite WMS CLI (glite-wms-job- commands)

• The UI test suite has been updated• The test suites implement the concept of efficient testing,

which represents an attempt to maximize the coverage and utility of the test suite while minimizing its size and complexity

• The test suites are used for certification of gLite patches and updates, and can also serve to verify UI installations (e.g. afs UI), search for new bugs, and other.

• TWiki-based documentation exists• The author is still available to support these packages

Efficient testing of Command Line Interface of Grid UI

D. Zaborov

https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#User_Interface_UI https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#WMS_CLI https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#Data_Management_DM

Five reasons to test Grid UI

1. UI is an essential component of the Grid2. A Grid user may not need every Grid service, but

(s)he can not do without a UI3. A typical Grid UI is a rather sizeable program

complex that may contain more program components than certain Grid servers

4. UI functions, in particular command-line interfaces (CLI), are offten used within software packages (e.g. LHC experiments software)

5. Grid users can install and run a UI on their machines, which may vary in OS version and configuration. UI is also offten installed in a shared disk area and then used from many workstations, which again may differ in configuration. The functionality of the UI distribution should be ensured in all cases whenever possible

Tar UI in afs/shared disk

UIUI

UI probably needs even more elaborate testing than some Grid services. Special

effort should be made to detect configuration problems and software

conflicts

PC

CLI A

C++ API A

…

Service type A

Service type B

Service type

X

CLI X

Java API X…

…

…

A generalized approach to CLI testing

• Test target: a command line interface (CLI) with many commands and options

– It may be impractical or impossible to test every combination of various options by hand or even with an automated script

• Aim: maximize the probability to find an important bug/problem while minimizing the size and complexity of the test suite

– It is impossible to find all bugs and problems with a single test suite, however the most critical problems must be detected as quickly as possible

– Compact test suite is more easy to maintain and work with. Compact test report is more likely to be examined “by hand”

• Strategy: choose a minimal subset of possible use cases, which would be sufficient to detect the most common types of problems and conflicts

– The test cases should be organized into a series of test scripts in a reasonable manner (e.g. a “file replication test” would test lcg-rep command and all its options)

Part I: test every command

without any options

Part II: test every option of

the most used/important

commands separately

Part III: try all the options (or at

least most of them)

simulataneousely

The most typical use case!

Another typical use case. Such problems as “option does not work” or “option does not work without other option” can be detected. Options which are offten used in combination can/should be tested together. Options which are very unlikely to interact with each other can also be tested in one go.

This allows to detect conflicts between options (like “two options do not work together”). Complex types of conflicts, like “two options work together only when a third one is not given”, may not be detected though (but they are less likely to occur and to be faced by a user)

Specificity of Grid UI as a test target and ways to proceed

• UI is basically a set of client tools and libraries– Need to test various CLI’s (the most used), as well as API’s (C++, java, python, …). A typical CLI makes use of

the corresponding API. Thus API is indirectly tested by CLI tests.– Unlike for the Grid services, there is no need to run UI tests continuousely. However one may need to repeat

testing after major updates or configuration changes• Every sensible function of Grid UI involves the use of Grid services

– It is difficult to decouple client-side testing from server-side testing. But there are at least two reasons to test the client part separately: 1) usually client tools are distributed separately from server packages; 2) administration of UI machines is offten done independently from service administration

– Way to go: be sensitive to errors which could be caused by client software malfunction, but tolerate all kinds of errors occurring for sure “on the other side” (like job failure on a Worker Node)

• A great majority of problems with the UI are connected with configuration– Need special tests to detect configuration problems– Does command/library exists and can be found? detect problems with installation/shell environment– Does command run at all (e.g. with --version or --help)? detect problems with libraries, version conflicts

UI

CLI A

CLI B

API A Service A

resource x

resource y

test ignore

…

Testing gLite UI

UI

OS

Shell environment

gLite component

Grid service

test

Test suite gLite componentgLite

component

Test suiteTest suite

Grid serviceGrid service

test

Gen

eral

tes

tsgL

ite-s

peci

fic

clie

nt-o

nly

test

s

gLite

-spe

cific

tes

ts

invo

lvin

g G

rid

serv

ice

usag

emight be useful for any other, even non-Grid, node!

Check shell environment for non-existing directories in the PATH, etc. Different types of shell should be addressed (csh, bash)Check that ntp daemon is running - proper clock synchronization is important for many types of client-service interactionCheck gLite shell variablesCheck that every gLite command metioned in the User Guide can be foundAlso check that documentation (man page) is supplied with every command Check that the gLite libraries are in placeTry to run the commands with the simplest options (--version, --help, …) Security subsystem tests (voms-proxy-init, etc)Information system testsWMS CLI test suiteData Management CLI test suite…

Proved useful!