Upload
emory-hart
View
220
Download
2
Tags:
Embed Size (px)
Citation preview
CITLAB: a Laboratory for CombinatorialInteraction Testing
Angelo GargantiniUniversità di Bergamo - Italyhttp://cs.unibg.it/gargantiniJoint work with Paolo VavassoriIn collaboration with Andrea Calvagna Università di Catania
Workshop on Combinatorial Testing (CT) @ ICST 2012 April 17, 2012
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
lSome open issues in CTuser perspective
Still missing• A common abstract language • with a precise semantics for concepts like constraints, seeds …
• A standard concrete syntax/grammar • or exchange format for tools: to guarantee interoperability
• Limited assistance in writing the models• generation tools do not offer any editing capabilities (only a
grammar and a parser) and are rarely integrated in any IDE for programming or design.
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
lSome open issues in CT researcher perspective
• No standard language Comparison among tools and algorithms is difficult• Conceptual point of view: some features may be supported in an
approach but not in another or different meanings• Practical: every comparison is quite unreliable since every user must
redefine in its own language and tool the examples
• No help in defining new algorithms and tools• In order to experiment a new test generation algorithm, a researcher
should define a proper grammar and a parser, develop the libraries to manipulate the model data, and translate the benchmarks
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
CITLAB in brief
(Domain Specific) Language 1. A rich abstract language with a precise formal semantics2. A concrete syntax with a well defined grammar by XTEXT3. A textual editor integrated in the eclipse IDEA Framework4. based on the Eclipse Modeling Framework (EMF) which
provides tools and run-time support to manipulate combinatorial problems in Java
5. A rich collection of Java utility classes and methods6. For introducing new test generation algorithms7. For introducing code translators for importing and
exporting models and tests
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
A DSL for CIT problems
• Domain-specific languages (DSLs) are languages tailored to a particular problem domain.• They offer substantial gains in expressiveness and ease of use
compared with general-purpose programming languages • CIT can be considered a specific application domain for which
developing a domain-specific language is worthwhile.• DSL development is supported by several tools and technologies.
• XtextXtext is a framework for development of domain specific languages. It covers all aspects of a complete language infrastructure, from parsers, over linker, compiler or interpreter to fully-blown top-notch Eclipse IDE integration.
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
.ecore(metamodel)
Language project
.xtxtGrammar
Generator
Constraints
Antlr Parser EditorSyntax coloring
Manual (required)
Manual(optional)
Editor Project
Generated
Java API
Language Model
Outline
Content Assist
Template
Formatting…
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
A DOMAIN SPECIFIC LANGUAGE FOR CIT PROBLEMS
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
CITLAB Language in a glance
Model ModelTypes:...endParameters:...endConstraints:# ... #endSeeds:# ... #endTestGoals:# ... #end
Parameters
Types
Constraints
Seeds
Test goals
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Parameters and their types
• To describe a combinatorial problem would be sufficient to specify the number of variables and their cardinality.
• CITLAB language forces the designer to name parameters and to specify their types by listing all the values in their domain.
Parameters: Boolean emailViewer; Range textLines [ 25 .. 30 ]; Enumerative display { 16MC 8MC BW };end
1. Enumerative for parameters that can take a value in a set of symbolic constants.
2. Boolean for parameters that can be either true or false.
3. Numerical value for parameters that take any value in an integer range.
Choice: explicit parameter names to facilitate the modeling of real systems and to ease the specification of constraints and seeds
From ACTS
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Named Types
• Types can be defined with their name in the Types section to be used in parameters declaration• Instead of an “anonymous” type.
Types: EnumerativeType cameraType { 2MP 1MP NOC };endParameters: Enumerative rearCamera : cameraType; Enumerative frontCamera : cameraType; ...end
Advantages: the use of named parameter types to make more compact and more maintainable the models in case many parameters share the same domain.
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Constraints
• In CITLAB, we adopt the language of propositional logic with equality and arithmetic to express constraints
• General Form (GF) constraints• propositional calculus and Boolean operators
a or b => c and d
• equality and inequality emailViewer = true or frontCamera ~= NOC
• arithmetic over the integers • relational and arithmetic operators for numeric terms
textLines >= threshold + 10
• A test t satisfies a constraint C or it is a model of C iff C is true by substituting each parameter pi in C with the value of pi in t
Constraints Semantics
• A test t is valid if it is a model of all the constraints , i.e., or equivalently
• Challenge: finding “valid” “complete” test suites• Because of constraints, some combinations are not coverable• A test suite must cover “only” all the coverable combinations• Finding a (if any) model is an NP problem = SAT problem
• Checking if a test satisfies a constraint is however linear• Checking if a combination violates the constraints is linear
• Challenge: assuring constraints consistency• constraints are consistent if there exits at least a test that satisfies
them, i.e., • Consistency checking is again a SAT problem• FUTURE WORK: use a SAT/SMT solver for this
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
GF vs forbidden combinations
• Most CIT that support constraints, allow only the use of forbidden combinations or tuples
• A forbidden combination would be translated in our model as a not statement. • a forbidden pair x = a; y = b would be represented# not (x = a and y = b) #
Advantages:• Much more compact than forbidden tuples
Example: if b {x1 … xn }“a = x => b = xk” would need n-1 forbidden tuples
• More expressive (whilst in a finite domain …)• More similar to the natural language• No need to distinguish between implicit and explicit constraints
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Seeds
• The testers can also force the inclusion of their favorite test cases by specifying them as seed tests. • They can be tests generated by other criteria• Critical complete combinations
• The seeds must be included in the generated test set without modification
• CITLAB considers only complete seeds, i.e., seeds that assign a valid value to each parameter
# frontCamera==1MP, display==16MC, emailViewer=true, rearCamera==2MP, textLines=30 #
• Partial seeds? Critical partial combination the tester would like to be included in the test suite
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Test Goals
• Critical situation that must be tested • Predicates that must be covered by some tests• Test goals can be again in GF (as the constraints):// at least one camera is missing and the display has at least threshold lines
( rearCamera==NOC or frontCamera==NOC) and textLines>=threshold
• Partial combinations // both rearCamera and frontCamera are NOC and display is 16MC
rearCamera==NOC and frontCamera==NOC and display==16MC
• Remarks:• Checking feasibility of test goals is an NP-complete problem, since it can be
reduced to a SATisfiability problem• Checking if a test covers a test goal is however linear
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
CITLAB EDITOR
DEMO
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Editor features
• Syntax Coloring• Content Assist• Template Proposals• Rich Hover• Rename Refactoring• Quick Fixes• Outline
• Folding• Hyperlinks for all
Cross References• Find References• Toggle Comment• Mark Occurrences• Formatting
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Example, model validation
• XTEXT provides several levels of validation for the defined language• the user can specify additional constraints for the model by
providing validation fragments
Example: No seed can violate any constraint
Note: • checking if a seed violates a constraints is decidable in a
linear time
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
AN EXTENSIBLE FRAMEWORK FOR CIT
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
CITLAB: as a extensible framework
CitLab framework
Random generator
• For the definition and implementation of test generators and a set of exporters/importers to and from other languages
• Based on the eclipse plugin development environment
• Goals• to foster tools interoperability• to favor reuse of artifacts• to allow fair comparison
Casaexporter
generator exporter
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Extension points and extensionsCitLab accepts new
contributions as pluginsby defining extension points
Contributors add new plugins by implementing extensions• Declare the extension in .xml• Implement an Interface
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
CITLAB: extension pointsExtension Point Interface/Methods
TestGenerator ICitLabTestGenerator
TestSuite generateTests(Model m, int nWise)boolean acceptConstraints(List<Constraint> c)boolean acceptSeeds(List<Seed> s)boolean acceptTestGoals(List<TestGoal> tg)
Exporter ICitLabExporter
void export(Model m)boolean acceptConstraints(List<Constraint> c)boolean acceptSeeds(List<Seed> s)boolean acceptTestGoals(List<TestGoal> tg)
Importer ICitLabImporter
Model import(Reader r)
TestSuiteExporter ICitLabTestSuiteExporter
void save(TestSuite ts, String file)
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Using Model to Text or Model to Model
• MDE approach: a possible way to define importers and exporters is to use • Model to Text (M2T) or • Model to Model (M2M) transformations
• For instance, Xtend2 for CASA exporter
def toCasaCode(Model sm,int n){ ''' «n» «sm.parameters.size» «FOR param :sm.parameters»«param.getSize» «ENDFOR» '''}
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Test Suite meta-model and Java utils
• a simple meta-model for tests and test suites
• We have developed several classes and methods capable to perform routine tasks like • generating all the test requirements for a given strength, • checking if a test or a seed violates some constraints• checking if a test goal is covered by a given test suite• …
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
CitLab site and Licence issues
• All the code is available at http://code.google.com/a/eclipselabs.org/p/citlab/
• CitLab is released under the Eclipse Public License (EPL) • an open source software license • used by the Eclipse Foundation • a business-friendly free software license
• copyleft • but weaker than other, like GNU General Public License (GPL)
• can use, modify, copy and distribute the work and modified versions, in some cases being obligated to release their own changes• CitLab Plugins can be commercial (distribute as binaries)• Any modification of the CitLab framework must be open sourced
• The EPL is approved by the Open Source Initiative (OSI) and is listed as a "free software license" by the Free Software Foundation (FSF).
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Status and Future work
• Adding new plugins is easy (as student projects)• Paolo Vavassori has a part-time contract for 1 year
• Already implemented• Generator
• Random with seeds support• IPO and AETG work in progress
• Model exporter• To CASA (constraints still in progress)
• Test suite exporter• To excel /CSV file
• New planned features• Language, e.g. output for oracles, arrays, …• Meta-algorithms (like: run 50 times and take the best)• Test suite postprocessor, e.g. remove unnecessary test cases
Ange
lo G
arga
ntini
- Ci
tLab
: a L
abor
ator
y fo
r Com
bina
toria
l
Objectives and conclusions
CitLab Language & editor• Establish a CIT “language” with a precise semantics• New concepts and constructs?
• Exchange models and benchmarks• CIT “knowledge base”
• A user friendly textual editor integrated with the eclipse IDECitLab plugin-based environment • defining new generators • to foster contributions from new parties
• making comparison easier and more reliable