Middleware Testing Framework

Outline

• Exhaustive testing v. Assurance Testing• What do we test?• How often do we perform out tests?• What do we use to schedule and perform

tests?• How do we view the results?• What to do with the results?

Exhaustive v. Assurance Testing

• Exhaustive testing:"seeks to verify the behavior of every aspect of an element, including all permutations. For example, exhaustive testing of a given user command would require testing the command with no options, with each option, with each pair of options, and so on up to every permutation of options." Exhaustive testing or developing tests for all requirements of a standard or specification can take many staff YEARS and be prohibitively expensive. In some cases it is impossible to test all of the possible test cases in a reasonable amount of time.

Exhaustive v. Assurance Testing

• Thorough testing:"seeks to verify the behavior of every aspect of an element, but does NOT include all permutations. For example, to perform thorough testing of a given command, the command shall be tested with no options, then with each option individually. Possible combinations of options may also be tested."

What Do We Test?

• Services– Are they alive?– Are the providing their expected behavior– Do they support a typical user session

• Scripts– Do they throw the proper errors for specific

instances of invalid scripts– Do they submit valid scripts– Are they available and in the proper locations?

What Do We Test?

• Resources– Are they available– Can we login– Can we stage data in properly– Is the file system full– Do they run jobs to completion– Do they properly checkpoint jobs– Do they properly kill jobs– Do they send out the appropriate emails– Do they stage data out properly

How Often Do We Perform Tests

• Test entire infrastructure daily?• Test services several times a day?

What Do We Use To Test?

• External services,• API’s• Command line utilities• Cron jobs

How Do We View the Results

• Using CruiseControl…

http://portal.gridchem.org/status

– build management system– generates reports– aggregates data– keeps historical information– open source available on sourceforge.

What Do We Do With The Results?

• Good question• Reporting• Help answer ticketing issues• Improve our QoS• Improve our software engineering practices.• Advertise and Market GridChem and the

CCG!!!

Questions?