1

Verification of Semantic Web ServiceAnnotations Using Ontology-Based Partitioning

Khalid Belhajjame, Suzanne M. Embury, and Norman W. PatonSchool of Computer Science, University of Manchester

F

Abstract—Semantic annotation of web services has been proposedas a solution to the problem of discovering services to fit a particularneed, and reusing them appropriately. While there exist tools that assisthuman users in the annotation task, e.g., Radiant and Meteor-S, nosemantic annotation proposal considers the problem of verifying theaccuracy of the resulting annotations. Early evidence from workflowcompatibility checking suggests that the proportion of annotations thatcontain some form of inaccuracy is high, and yet no tools exist to helpannotators to test the results of their work systematically before theyare deployed for public use. In this paper, we adapt techniques fromconventional software testing to the verification of semantic annotationsfor web service input and output parameters. We present an algorithmfor the testing process, and discuss ways in which manual effort from theannotator during testing can be reduced. We also present two adequacycriteria for specifying test cases used as input for the testing process.These criteria are based on structural coverage of the domain ontologyused for annotation. The results of an evaluation exercise, based ona collection of annotations for bioinformatics web services, show thatdefects can be successfully detected by the technique.

Index Terms—Semantic web services, semantic annotations, test ade-quacy, specification-based testing, model-based test generation.

1 INTRODUCTIONSemantic annotations of web services have been pro-posed as a (partial) solution to web service discovery andcomposition [5], [4]. These annotations relate the variousservice elements (i.e. operations, inputs and outputs)to concepts in ontologies describing their semantics,form and role. For example, an operation which takesa protein sequence as input and performs a search forthe most similar protein in some given database might beannotated with the task concept SimilaritySearch. Its sin-gle (string-valued) input parameter might be annotatedwith the concept ProteinSequence, and its output with theconcept ProteinAccessionNumber.

Such semantic annotations can be used by tools toe.g., assist in service discovery and composition [16],[17]. Such tools can only be of value if semantic anno-tations accurately reflect web services’ semantics. Ourexperience using data flow connections between ser-vices in workflows to check compatibility of annotationshas revealed that this is frequently not the case [2].Manual annotation of services is a time-consuming taskthat requires expertise in the application domain, in itsrepresentation in the specific ontology being used, and

in the task supported by the web service. Where theannotations are supplied by the web service providers,one would reasonably expect them to be of good quality,in the sense that the chances that they suffer from errorscan be low. However, the annotation task is generally notintegrated within the web service development lifecycle,and, as a result, web services are often annotated bythird parties. For example, web service annotations inregistries, such as Biocatalogue1, are provided in themajority of cases by the users of web services or curatorswho are hired to perform the annotation task, and not bythe service providers. In the typical case, the annotatorwill have been involved with neither the design of thedomain ontology nor the implementation of the webservice, and there is therefore considerable scope formistaken interpretations of concepts and behaviours inselecting annotations. Despite this, there is a dearth oftools and methods that can assist in the verification ofannotation accuracy. Based on discussions with annota-tors working on the myGrid project2, peer inspection ofannotations is the main means of verification in practicaluse, supplemented with feedback from users after theannotations have been published.

In this paper, we explore how techniques from soft-ware testing can be applied to the problem of verifyingsemantic annotations of web service parameters. Soft-ware testing is a well established discipline that hasbeen a major research topic in software engineeringduring the last three decades, techniques from whichhave proved to be effective in practice for verifyingthat the behaviour of a software program conforms withits specification [18]. A software program, e.g., a webservice, is tested by using a test suite composed of acollection of test cases, each of which specifies data val-ues for feeding the software execution and the outputsexpected as a result according to the specification. Thesoftware is executed using the input values specified bythe test cases. A defect is found if the outputs deliveredas a result of the software execution are different fromthose expected by a test case.

1. http://www.biocatalogue.org2. http://www.mygrid.org.uk

2

We adapt the above technique for verifying the an-notations of web services. We begin by examining thesemantics of annotations on service parameters, andderive from this the form of a test case for parameterannotations (Section 2). These semantics provide us withthe basics of an oracle for automatic test case generation(Section 3). An exhaustive test that uses every possibletest case can be expensive, and most of time is impossibleas the domains of the inputs can be infinite. Because ofthis, only a subset of possible input values is used to con-struct test cases. To ensure that this subset is representativeof the range of possible input values, thereby increasingits defect-detecting power, it is constructed carefullybased on test adequacy criteria. In this respect, wepropose two new adequacy measures, based on coverageof concepts in the ontology used for annotation, thatcan be used to automatically select the inputs needed toform representative test suites (Section 4). We assess theeffectiveness of the adequacy criteria by applying themto the verification of real-world annotations producedby a domain expert, and show that both approachesresult in much higher defect detection rates than whenthe same number of test cases are selected at randomfrom the input pool (Section 5). We compare our methodto existing proposals (in Section 6), and conclude byhighlighting our main contributions (in Section 7).

2 VERIFYING SEMANTIC ANNOTATIONS

Semantic annotations for web service parameters areeffectively high-level descriptions of the domains of theparameters that go beyond the usual data types foundin interface description languages (such as WSDL). Assuch, it should be possible to generate test cases basedon the information they contain, in much the same wayas is done in other forms of model-driven/specification-based testing (e.g. [9], [10], [24]). In these approaches,however, the specification is regarded as being the cor-rect description of the software’s behaviour, and thetest cases created are intended to reveal defects in thesoftware. In the case of semantic annotations, however,which are typically generated after the code has beenpublicly deployed, and by some third party not involvedin its creation, the software is considered to be de factocorrect and the test cases derived are intended to verifythat the specification accurately reflects its behaviour. Inaddition, we are not attempting to verify the completebehaviour of the web service. Instead, we are attempt-ing to verify a somewhat weaker proposition, namely,whether the concepts used to annotate the parametersare accurate descriptions of the sets of values that canlegally be input to or output by the service.

In the standard model-based test case generation ap-proaches, for a web service with n input parameters,we would use the specification model to generate thefollowing pieces of information for each test case:

• An n-tuple containing the values that should beused as inputs to the web service in one test run.

• A statement of the expected outcome of the testwhen these input values are given.

In order to generate test cases for semantic parameterannotations, therefore, we must show how to create boththese test case components based only on the ontologyconcepts that have been assigned to the web serviceparameters under test.

2.1 Generation of Inputs for Parameter Tests

Since web services are black-box software components,we must find a way to verify the input parametersemantics by executing the services and observing theresult. In conventional testing, we would be interestedin the actual data value returned by the web service forsome given input values, but in this context, as we havesaid, we are testing only whether the input is legal forthe service and not whether it leads to a correct result.How, then, can we determine whether an input is legalbased on observation of the execution of the service?

In order to provide one possible answer to this ques-tion, we introduce the notion of acceptance of an inputparameter by a web service operation. The web serviceinterface standard allows programmers to clearly distin-guish normal termination of an operation (in which aresult message is returned, for example) from abnormaltermination (in which an exception is thrown). Therefore,we can say that an input (or collection of inputs) isaccepted by an operation only if, when supplied withthe input(s) as parameters, the operation terminates nor-mally. This gives us a very simple operational means ofdistinguishing web service behaviour with valid inputsfrom that with invalid inputs. Note, however, that inpractice, unfortunately, it is more difficult to interpretboth normal and abnormal terminations of web services.In the case of abnormal terminations, there are severalother factors that might cause a web service to exit withan exception, besides being supplied with an invalidinput. For example, an e-bookstore may report that thebook is out of order. Such a termination should not beinterpreted as abnormal. Also, the server hosting theweb service might be temporarily inaccessible, or theservice may be attempting to manage its own load byrefusing requests when its load gets too high. In somecases, problems will go away if the tests are run again,and the user need not be bothered with them. However,in others, the only safe way to interpret the results ofthe test is to ask a human user to investigate. Similarly,a service execution which seems normal, i.e., does notthrow any exception, may be unsuccessful. Althoughthe web services standards provide proper exceptionmechanisms for flagging errors to the caller, many webservice implementations make no use of them. This iscommonly the case when a web service is rapidly imple-mented by wrapping existing software implemented in aprogramming language that did not support exceptionhandling. Such programs typically report errors to theuser by returning an empty string, or a string value that

3

contains an error message. For example, the getUnipro-tEntry web service provided by the DDBJ3, returns anempty message when the value used as input is invalid.Where the forms of these error messages are known,we can write string matching functions that distinguishbetween a normal and an abnormal termination evenwhen no exception has been thrown.

We can see that to test a semantic annotation foran input parameter, we should generate a collection oflegal and illegal values for the parameter, and create testcases which look for normal and abnormal terminationrespectively. For a given parameter ip annotated withconcept c

ip

, the set of legal values are those which are inthe extension of c

ip

and the set of illegal values are thosein the extension of ¬c

ip

(i.e., any value in the extension ofany concept that is not equivalent to c

ip

and which is notin the extension of c

ip

). Thus, to generate candidate inputvalues, we need a pool of data values that are annotatedwith the ontology concepts to which they belong.

Since semantic types are typically more complex thanstandard data types, the task of generating values forthem is correspondingly more challenging. Althoughsome finite numeric ranges or simple enumerated typesmay be present as parameters for some operations, ingeneral we cannot expect to be able to provide a set ofbuilt-in data generators that will cover the majority ofcases, as we can in software testing. For certain semantictypes, it may be possible for the user to indicate acomputational procedure by which candidate instancescan be obtained. For example, a large and representativecollection of ProteinSequence instances can be createdby querying one of the major public protein databases(such as Uniprot4). This may be a satisfactory solutionfor semantic types that occur frequently in operationparameters, but it demands too much effort from theannotator to be a good general solution.

An alternative source of annotated instances doesexist, however, namely, workflow provenance logs [6].A provenance log is effectively an execution trace that isrecorded by the workflow management system duringworkflow enactment. Most importantly for our pur-poses, these logs contain copies of the intermediate datavalues that were produced by the execution of eachcomponent service in the workflow. Every data item thatwas passed as an input parameter to some componentservice, or returned as an output, is recorded in the log.Where those services have been semantically annotated,the logged data values are also tagged with the semantictype of the parameter with which they are associated.

By trawling these workflow logs, we can collect an-notated instances for use in test case generation. Note,however, that the annotations found in the provenancelogs may not be entirely accurate. As in object-orientedprogramming, if an input parameter has a semantic typeof c then it may legitimately take instances of c or any

3. http://xml.nig.ac.jp4. http://www.uniprot.org

of its subconcepts as values. Thus, the annotations givento the data values in the provenance logs may often besuperconcepts of the correct annotations, and thereforesomewhat inaccurate. Fortunately, however, this doesnot affect the validity of their use in test case genera-tion. Consider, for example, an operation which takesa Sequence but which is supplied with an instance ofProteinSequence (a subconcept of Sequence) as an input atrun-time. This instance will then be mistakenly tagged asa Sequence in the provenance logs, and may be suppliedas a possible test input whenever Sequence instances arerequired. But, since the instance actually is an instanceof its tagged class, no serious consequences arise.

It is worth stressing that only workflows with servicesthat have annotations that are known to be correctare used. Moreover, the instances that are collected bytrawling provenance logs are checked by a human userbefore they used in the generation of test cases.

So far, we have focused on the values to be usedfor constructing the test cases for input parameters.Regarding the test cases for output parameters, we ex-ploit an obvious source, namely the results generated bythe service operation in question when invoking themusing the (legal) input values from the pool of annotatedinstances.

2.2 Determining Expected Outcome for ParameterTests Under the Weak SemanticsTo complete the test case for a given operation param-eter, we must be able to determine what the expectedoutcome of the case should be, if its annotation is anaccurate description of the legal values for that param-eter. For this, we need a clear statement of the intendedsemantics of parameter annotations. Unfortunately, theliterature on semantic web services has not reached aconsensus on exactly what the meaning of a semanticannotation for an operation parameter is. Two alternativesemantics can be gleaned from the literature. Unsurpris-ingly, the clearest statement of intended semantics canbe found in the proposals for languages for specifyingsemantic annotations (such as SAWSDL5, WSML6 andOWL-S7). The specifications of these languages state thatall valid inputs for an annotated parameter must belongto the domain concept with which it has been annotated.However, it is not necessary for the operation to acceptall such instances as valid (or, at least, such a condition isnot explicitly stated by the authors of these proposals).

To illustrate this point with an example, let us returnto the SimilaritySearch operation mentioned earlier, thesole input parameter of which is annotated with theProteinSequence concept. This annotation should be readas indicating that no instances of any concepts that aredisjoint with ProteinSequence should be accepted by theoperation; so no NucleotideSequence instances should beaccepted, nor any Organism instances. But, there may

5. http://www.w3.org/TR/sawsdl/6. http://www.wsmo.org/wsml/7. http://www.daml.org/services/owl-s/1.0/

4

also legitimately be instances of ProteinSequence thatcannot be accepted by the operation.

We call this interpretation of the annotations the weaksemantics, since it does not place very stringent require-ments on which instances the operation can accept. Sim-ilarly, under the weak semantics, the instances that aregenerated by an operation output belong to the conceptused for annotating that output. However, the output isnot required to produce every possible instance of theconcept used for annotation. With the following nota-tion, we can define this semantics formally. An operationop has a set of input parameters given by the functioninputs(op) and a set of outputs given by outputs(op). Aparameter is denoted by a pair hop, pi where p is thename of the parameter and op is the operation to whichthe parameter belongs. The set of all input parameters isINS and the set of all outputs is OUTS. If ✓ is the setof concepts {c1, . . . , cn} in our domain ontology, thenthe function: domain : INS [ OUTS ! ✓ returns thedomain annotation for any given parameter. Given afunction hasInstances(), which takes a concept and returnsthe set of all instances of that concept, we can define acorrect parameter annotation under the weak semanticsas follows.

Definition 2.1: An input (resp. output) parameterhop,pi is annotated correctly under the weak semanticsiff:

• there exists an instance i 2hasInstances(domain(hop, pi)) that is accepted(resp. generated) by op, and

• no instance j 2 hasInstances(¬domain(hop, pi)) ex-ists that is accepted (resp. generated) by op.

Based on this definition, we can see that if an op-eration accepts (resp. generates) a particular instanceof a concept that is disjoint with the annotation c forthe relevant input (resp. output) parameter, then theannotation is incorrect. However, if an instance of c isnot accepted (resp. generated) by the operation, then wecannot conclude that the annotation is incorrect.

As explained earlier, in the case of input parameters,we use operation acceptance as a means for determiningvalidity based on the execution of the web service oper-ation in question. We need a similar operational meansfor output parameters. By selecting legal input valuesfrom our pool of annotated instances, we can create aset of output values produced by the operation. How-ever, to determine the result of the test case, we needto determine to which semantic concept the instanceproduced belongs. This is a much harder task than wefaced when generating input parameter test cases, whenwe were given a concept and had to locate instances inits extension. In the case of output parameters, we havethe reverse task. For example, given a string value, wemust determine whether it is an instance of DNASequenceor GeneIdentifier or LabName.

Annotating an instance, that is discovering the seman-tic type of an instance, based solely on its content is, ingeneral, a difficult task. One (last resort) option would

be to rely on the domain expertise of the annotator, andto ask him or her to classify the output values producedby the web services. However, it would obviously bepreferable if some automatic means of classifying outputinstances could be found.

In fact, an automatic mechanism does exist. Our dis-cussion of the requirements for testing input parameterspoints to a resource of software components able toperform this classification task: namely, the annotatedweb services themselves. Consider, for example, a webservice with an output that is annotated with the Ge-neOntologyID concept. If we assume that the annotationsof service inputs have a strong semantics (see Section2.3), then any web service with an input parameterannotated with this same concept can, in theory, act asa means of testing whether the output value belongs tothe concept or not. For instance, we can potentially usethe GetGOTerm8 web service to verify the semantic typeof any values claiming to be instances of the GeneOntol-ogyID concept. If GetGOTerm accepts the output value,then we know it is a valid gene ontology ID.

A test case for an output parameter annotation, there-fore, looks similar to that for an input annotation. Wemust provide a collection of values to be used as theinputs to the operation under test, in order to generatea sample output value. We then define the expectedresponse in terms of normal or abnormal terminationof a web service. The main difference is that the testcase involves the execution of a composition of twooperations, rather than just one operation.

All this, however, depends upon the rather rash as-sumption that the input annotations are correct. Since wehave evidence to suggest that quite a high proportion ofannotations are inaccurate to some degree or other [2],this approach to verifying output parameters shouldonly be used with web services with trusted inputannotations. This suggests a phased approach to testing,in which the annotator first concentrates on verifyingthe input annotations for a batch of web services, andon correcting any errors found, before beginning totackle the correctness of output annotations. Anotheradvantage of this phased approach is that we can collectand record the details of the output values generatedwhile testing the input parameters, and thus save sometime in the output parameter testing phase.

2.3 Determining Expected Outcome for ParameterTests Under the Strong SemanticsThe annotation of an operation input is said to have astrong semantics, if the operation accepts any instancethat belongs to the concept used for annotating theinput. This alternative strong semantics for input pa-rameter annotations is implied by the requirements ofseveral applications in which annotations are used, inparticular service discovery and workflow composition.Consider, for example, tools that assist designers in the

8. http://www.inab.org/dproxy/inb.bsc.es/getGOTerm?wsdl

5

composition of workflows, e.g., Biomoby Taverna plug-in [25] and Galaxy [7]. Given a service operation in anincomplete workflow, such tools suggest to the designerthe service operations that can be composed with it,in order to move towards the completed workflow.Specifically, such tools use the semantic annotations ofoutput parameters to search for services that are ableto consume the values output by the service already inthe workflow. The implication here is that the suggestedservice operations are able to consume any instance ofthe concept used to annotate the output of the precedingservice in the workflow. The same implication can bemade from tools that diagnose parameters connectionsto detect mismatches [1].

We can formally define correctness of annotationsunder this strong semantics, as follows.

Definition 2.2: An input parameter hop,pi is annotatedcorrectly under the strong semantics iff:

• every instance i 2 hasInstances(domain(hop, pi))is accepted by op, for every assignment of validinstances to all other input parameters of op, and

• no instance j 2 hasInstances(¬domain(hop, pi)) ex-ists that is accepted by op for some assignment ofinstances to all other input parameters of op.

Note that while the annotations of input parametersspecified using languages such as OWL-S and WSMOhave a weak semantics, these languages offer a mecha-nism, namely preconditions, using which the semanticsof annotations specified can be upgraded from weakto strong. Specifically, using WSMO, for example, everyoperation op can be associated with a precondition prec,that only holds when the values bound to the inputsparameters of the operation op are accepted by op.

In the rest of this paper, we assume that the inputannotations has a strong semantics, since it is the seman-tics adopted in practice as illustrated in the workflowcomposition example presented above. That said, wediscuss the conclusions that cannot be made when theannotations have a weak semantics.

Regarding output parameters, it does not seem rea-sonable to insist that an operation with an output anno-tated with a concept c

op

must be capable of outputtingevery possible instance of c

op

. To illustrate this using anexample, consider a service operation that produces abiological sequence. It is neither reasonable nor practicalto require such an operation to be able to produceevery possible sequence. Instead, service operations thatproduce outputs of that kind are confined to biologicalsequences that are either stored in a public databaseor in-house labs. Also, the set of output instances areusually reduced to the subset of the instances of c

op

that are output given valid input parameters for theoperation. Given the above reasons, we consider onlya weak semantics for output parameters.

The above definitions formalise the sets of valid andinvalid inputs implied by the annotations, and so pro-vide us with a means of deducing the expected outputfor a given test case (in terms of normal or abnormal

termination of the operation). In the case of input param-eters, if the annotations have a strong semantics, then wecan be more confident in interpreting the results of thetest than we could with the weak semantics. Here, if aninstance of a concept c is not accepted by an operationfor an input parameter that is annotated with c, then weknow that the annotation is definitely incorrect, since allinstances of c should be valid for this parameter.

Based on all this, we can now specify the full al-gorithm for testing semantic annotations for input andoutput parameters.

3 ANNOTATION VERIFICATION APPROACHWe start this section with an overview of the lifecycleof semantic web service annotations. We then presentthe details of the phases that constitute the annotationverification process.

3.1 The Lifecycle of Semantic Web Service Annota-tionsFigure 1 shows that the lifecycle of semantic annotationsof web services can be decomposed into three mainsteps: annotation, verification and use.

In the first step, Annotation, web services are annotatedeither manually, e.g, using Biocatalogue9 or Radiant [12],or semi-automatically using tools such as Meteor-S [20],APIHUT [11], KINO [21]. These tools provides the userwith suggestions for terms that can be used for anno-tation, and which are inferred using existing schemamatching and machine learning techniques. Detailedinformation about the above annotation tools is givenin the related work section.

In the second step, Verification, semantic annotationsare verified to identify potential defects. A detaileddescription of this step is given in Section 3.2.

In the the third step Use, curated semantic annotationsare deployed for public use. In particular, semantic an-notations are used to discover and compose web servicesusing tools such as Galaxy10, Meteor-S [20], APIHUT [11]or KINO [21]. Detailed information about these tools isgiven in the related work section.

Now that the lifecycle of semantic annotation of webservices has been presented, we can proceed to thedetails of the verification method.

3.2 Verification ProcessThe phases that constitute the verification process areillustrated and numbered in Figure 1. The verificationprocess starts by generating a pool of annotated in-stances (phase 1). Once the pool of annotated instancesis available, the test cases for input parameters are con-structed (phase 2), executed (phase 3), and the results areanalyzed to identify defects in inputs annotations (phase4). Test cases for the annotations of output parametersare then identified (phase 5) by exploiting the result ofexecution of the test cases for input parameters. The test

9. www.biocatalogue.org10. http://http://galaxyproject.org

6

Fig. 1. Web Service Annotation Lifecycle and the Verification Approach.

cases for outputs are executed (phase 6), and analyzedto identify defects (phase 7). A curator can then correctannotations in the light of the defects discovered (phase8). We present, in what follows, each phase in detail.

Phase 1: Generating a pool of annotated instances. Inthe first phase, a pool of annotated instances is generatedusing workflow provenance logs. Workflow provenancelogs are not used directly to verify parameters annota-tions, rather they are used as a source of instances ofontological concepts. In doing so, only provenance logsof workflows, that are composed of web services thatare known to have correct annotations, are used. Notealso that, this phase is not fully automatic; rather, it isdone under the control of a human who checks thatthe instances retrieved from the provenance logs belongto the concepts used for annotating the correspondingparameters.

It is worth mentioning that the use of workflow logsis not mandatory. Other sources of annotated instancescan be used. For example, where the ontology used forannotation is already associated with instances that areannotated using that ontology (or what is known asABox in the literature), then those instances can be usedfor constructing test cases.

Phase 2: Generating test Cases for Input Parameters.As mentioned earlier, we consider that the annotationsof operation inputs have a strong semantics, and that,if needed, operations are associated with preconditionsthat are used to enforce that semantics. We also discussthe conclusions that cannot be made when the annota-tion of input parameters have a weak semantics.

Generation of test cases for input parameters is anautomatic process. Once the user has specified the setof parameters and the semantic annotation registry tobe tested, as well as the location of the ontology usedfor the semantic annotations, generation is carried out asspecified in the algorithm in Figure 2. Interested readerscan find example test cases online11.Notice that the algorithm does not specify how legaland illegal values for each parameter (variable pool) areselected, since this is performed in the first phase. It isworth noting, however, that when the input parameterunder test hop, pi is associated with a precondition prec,then this predicate can be used to reduce the number oflegal values used for verifying the accuracy of the anno-tation of the input parameter to those satisfying the pred-icate prec. Specifically, a legal value v will be used for

11. http://img.cs.man.ac.uk/quasar/example test cases.php

7

Inputs ps

i

:= input params to be testedOutputs ts

i

:= empty test suiteBeginFor each operation parameter hop, pi 2 ps

i

pool := selected values from pool(should be a mixture of legal and illegal)For each v in pool

If v is legalSelect tuple of legal values lv forother input params of this operationsuch that the precondition of op holdseo := “normal

00 //expected outcome for lv [ v

If v is illegalSelect tuple of legal values lv forother input params of this operationeo := “abnormal

00

tc := hhop, pi, lv [ v, eoiAdd test case tc to ts

i

End

Fig. 2. Algorithm for Generating Test Cases for TestingInput Parameters

testing the annotation of hop, pi iff holds(prec, hop, pi, v),where holds(prec, hop, pi, v) is true iff prec is satisfiedwhen v is used as a value for hop, pi.

Phase 3: Executing test Cases for Input Parameters.When the test suite has been created, the test cases withinit are executed in turn, and the results are logged forsubsequent analysis (phase (3) in the overall approach).The steps required are specified by the algorithm inFigure 3. Existing tools such as SoapUI [8] can be usedin this phase, since they support large numbers of webservice invocations.

Inputs: tsi

:= test suiteOutputs: rs := a null tuple

inputLog := the results of execution of test casesoutputLog := the results delivered by

output parametersBeginFor each tc in ts

i

tc := hhop, pi

i, vs, eoiExecute operation op with input params vs

rs := result returned from op

If op terminated normally thenao := “normal”For each output param hop, p

o

i of opr := value for hop, p

o

i in rs

Add hhop, po

i, ri to outputLog

Elseao := “abnormal”

Add hop, vs, rs, aoi to inputLog

End

Fig. 3. Algorithm for Executing Test Cases for InputParameters

Notice that when executing an operation op using theinputs parameters vs, we may need to transform thevalues in vs to make them structurally compatible withthe representations adopted by op parameters. This isbecause different operation parameters adopt different

structures to represent the same value.

Phase 4: Interpreting the results of execution of testcases for input parameters. Once the test cases havebeen executed, the logged results from the input param-eter test cases are analysed. The main task at this point isto compare the expected outcome with the terminationstatus of the task, to determine which test cases haverevealed a defect and which have not. As hinted in thealgorithm in Figure 3, the result of the execution of atest case for an operation op is logged using tuples ofthe form:

hop, Iinput

, Ioutput

, statusi

Here, Iinput

is a set of pairs hhop, pi

i, insi

i where hop, pi

iis an input parameter of op and ins

i

is the instance usedto feed hop, p

i

i during the test case execution. Similarly,Ioutput

is a set of pairs hhop, po

i, inso

i where hop, po

i is anoutput parameter of op and ins

o

is the value of hop, po

ithat was obtained from the execution of op with the inputvalues given in I

input

. The final component, status, haseither the value normal or the value abnormal, describinghow the operation terminated.

We retrieve executions from the log using the function:

getExecutionsByParamValue : (INS [ OUTS) ⇥ I ! E

where I is the set of all instances and E is the set of alloperation log tuples just described. The function returnsall executions in which the given parameter took thegiven value.

In interpreting the results of each test case, thereare four possible situations to be considered. In thefollowing discussion, op is the operation executed duringthe test case, and hop, pi is the parameter being tested.The annotation associated with hop, pi is the concept c.

1) Valid Input/Normal TerminationIn this situation, an instance of c was supplied asthe value for hop, pi and the operation terminatednormally, as expected:

9 ins 2 domain(hop, pi),9 e 2 getExecutionsByParamValue(hop, pi, ins),

e.status = “normal 00

A test result of this kind presents no evidence forthe existence of a defect in the annotation, and it isnot reported to the user.

2) Invalid Input/Normal TerminationIn this situation, an instance of a concept that isdisjoint with c was supplied as the value for hop, pi.We would have expected an abnormal terminationfrom such a test case, but the operation has ac-cepted the supposedly invalid instance.

9 ins 2 ¬domain(hop, pi),9 e 2 getExecutionsByParamValue(hop, pi, ins),

e.status = “normal 00

This result is evidence of an error in the annotation.The parameter annotation may be over-specialised(i.e. the true annotation is one of its superclasses)

8

or it may simply be incorrect. This result should beflagged to the user.

3) Valid Input/Abnormal TerminationIn this situation, an instance of c was supplied asthe value for hop, pi but the operation unexpectedlyterminated abnormally.

9 ins 2 domain(hop, pi),9 e 2 getExecutionsByParamValue(hop, pi, ins),

e.status = “abnormal 00

This result is evidence of the presence of a defect inthe annotation of input parameter or the precondi-tion associated with op, if such a condition exists. Inthe case where the operation is not associated withany precondition, and the annotation of the inputhas a strong semantics, then this test is evidence forthe presence of a defect; it could indicate that theannotation for hop, pi is over-generalised (i.e. thecorrect annotation is a subconcept of c) or it maybe entirely incorrect.It is worth mentioning that in the case of the weaksemantics, a test run of this kind presents no evi-dence of any defect in the parameter’s annotation.

4) Invalid Input/Abnormal TerminationIn this situation, an instance of a concept thatis disjoint with c was supplied as the value forhop, pi and the operation terminated abnormally asexpected.

9 ins 2 ¬domain(hop, pi),9 e 2 getExecutionsByParamValue(hop, pi, ins),

e.status = “abnormal 00

A test result of this kind presents no evidence forthe existence of a defect in the annotation, and it isnot reported to the user.

As we pointed out, in practice, it is more difficult tointerpret both normal and abnormal terminations of webservices than the above characterisation suggests. In thecase of abnormal terminations, there are several otherfactors that might cause a web service to exit with anexception, besides being supplied with an invalid input.One of these is when an service operation with multipleinput parameters terminates with an error. This willoccur if any of the parameter values given is invalid;the problem may not be because of an error with theannotation for the parameter under test. To reduce thenumber of false positives of this kind reported to theuser, we consider only cases in which the operation hasalways failed when given this particular value for theparameter under test, regardless of the values of theother parameters. In other words, this kind of test failureis reported only when:

9 ins 2 domain(hop, pi),8 e 2 getExecutionsByParamValue(hop, pi, ins),

e.status = “abnormal 00

(The only change in the above expression compared withthe expression given in (3) is the quantifier in the secondconjunct.)

Phase 5: Generating test Cases for Output Parameters.To construct test cases for output parameters, we use theresults of execution of the test cases of input parameters.To do so, we examine the results of execution of test casesof input parameters, to filter out parameters for whichno output values were produced during the earlier inputparameters testing phase. Next, we further filter theparameter set to remove all those for which no annotatedweb service exists which takes as an input instances ofits semantic domain. That is, if we have a service thatproduces a ProteinSequence as an output parameter thatwe wish to test, then we must also have another servicethat takes as one of its input parameters (and preferably,the sole input parameter) instances of ProteinSequence.Where the web service used for testing has more thanone parameter, then we will need to select tuples oflegal values for other input parameters for the webservice as specified in phase (1) of the overall approach.The algorithm used for generating test cases of outputparameters is illustrated in Figure 4. Since we cannot relyon the input annotations being correct, we would furtherfilter out services for which many potential defects werelocated in the earlier testing phase.

Notice that this method for testing the annotations ofoutputs assumes that the annotations of the inputs of theweb services used for testing have a strong semantics.

Inputs: pso

:= output params to be testedOutputs: ts

o

:= empty test suiteBeginFor each operation parameter hop, pi 2 ps

o

os = output values for hop, pi from output logFor each ov in os

Find all services ss with input annotatedwith the same concept as hop, piSelect tuple of legal values lv forother input params of this operationeo := expected outcome for lv [ ov

tc := hhop, pi, lv [ ov, eoiAdd test case tc to ts

o

End

Fig. 4. Algorithm for Generating Test Cases for OutputParameters

Phases 6 and 7: Executing and interpreting the resultsof the test cases for output parameters. The executionof the constructed test cases is much the same as for theinput parameter tests (see the algorithm in Figure 5).Once the algorithm is executed, the results of executionof test cases are then analysed.

Phase 8: Diagnosing errors and correcting annotations.In this phase, the human curator examines the errorsdiscovered to filter out errors that are due to factors otherthan acceptance or non acceptance of values by serviceoperations. To amend incorrect annotations, the curatorexamines the ontology and chooses a different conceptfor annotation. In doing so, the verification results canbe exploited to facilitate the curator’s task. To illustratehow this can be done, consider that the annotation of an

9

Inputs: tso

:= test suite for outputsOutputs: outputLog := the results of execution of test casesBeginFor each tc in ts

o

tc := hhop, pi

i, vs, eoiExecute operation op with input params vs

rs := result returned from op

If op terminated normally thenao := ‘normal’

Elseao := ‘abnormal’

Add hop, vs, rs, aoi to output test logEnd

Fig. 5. Algorithm for Executing Test Cases for OutputParameters

Fig. 6. A Fragment of the myGrid Domain Ontology

input parameter i was found to be erroneous. Insteadof examining the whole ontology to identify the correctannotation, the curator can focus on a typically muchsmaller subset of the ontology composed of concepts,the instances of which were found to be accepted by theinput i.

Unfortunately, the above method of reducing the frag-ment of ontology displayed to the user to annotatethe parameter cannot be applied for the annotation ofoutput parameters. This is because, unlike the instancesof inputs, the instances of outputs are generally notannotated.

4 ADEQUACY CRITERIA

The test generation method described in Section 3 de-scribes a universe of possible test cases, divided intotwo kinds: those test cases that present valid inputs tooperations and those that present invalid inputs.

Just like in software testing [18], to construct a testsuite, we partition the domains of legal and illegal valuesinto partitions. In doing so we use the ontology usedfor annotation. The concepts within an ontology aretypically organised into inheritance hierarchies, giving amuch finer grained breakdown of the space of instancesthan the valid/invalid division that we have consideredso far in this paper.

This is illustrated in Figure 6, which shows a fragmentof the myGrid bioinformatics ontology [26]. Suppose thatthis ontology was used to annotate the single inputparameter of the SimilaritySearch web service with the

NucleotideSeq concept. This annotation could be incorrectin a number of ways. It could be an over-generalisationof the correct annotation if the web service is in fact onlydesigned to search for similar DNA sequences and notRNA sequence. It could be an over-specialisation, if theweb service is in fact able to search for similar sequencesof a variety of sorts, including protein Sequences andsequences that are neither protein sequences nor DNASequences. Or, the web service could have been designedonly to operate over protein sequences, in which case theannotation is simply incorrect.

Partitioning the input space for test generation accord-ing to the ontology classes holds out the promise of bothincreasing the diversity of the test suites that we generateand of providing more helpful information to the userregarding the pattern of failures and their possible causesin terms of annotation errors. In effect, the ontologyprovides us with a means of assessing structural cov-erage of test suites, thus leading to adequacy criteria,i.e., measures that can be applied to the test genera-tion process for increasing the representativeness of thetest suite constructed and, thus, increasing their error-detecting power. In the rest of this section, we presenttwo adequacy criteria, that exploit the ontology used forannotation, to partition the domain of legal and illegalvalues of operation parameters.4.1 Immediate-Sub-SuperconceptsOur previous work on inference and compatibilitychecking for semantic web services revealed that over-generalisation and over-specialisation of parameter an-notations are both prevalent in practice [2]. This firstadequacy criterion is designed to focus on the discoveryand diagnosis of exactly this kind of error, by requiringthat test suites include, for each parameter, test casesgenerated from:

• the semantic type c of the parameter (as defined bythe current set of annotations),

• all immediate sub-concepts of c, and• all immediate super-concepts of c.

4.2 All-Disjoint-ConceptsErrors that are not due to over-generalisation or over-specialisation in annotations are unlikely to be discov-ered using the partitioning strategy just described. Forexample, if an input parameter accepts instances that donot belong to the concept used for its annotation or tothe immediate superconcepts of that concept then thisannotation error will not be discovered using test casesgenerated by the Immediate-Sub-Superconcepts strategy.

Also, the extensions of the concepts obtained using thepartitioning strategy just described are likely to overlapand, therefore, do not form a partition in the mathe-matical sense. For example, the domain of nucleotidesequences covers that of DNA sequences. This is notpeculiar to our partitioning strategy. It is common, infunctional testing, for the domains obtained by partition-ing to overlap, e.g., this is often the case with domainsobtained by path-coverage partitioning [13].

10

However, where partitions do overlap there is anincreased possibility of generating redundant test cases.To force a more diverse and representative coverage ofthe space of instances, we can specify an adequacy cri-terion that requires the test suite to contain inputs fromeach concept in the ontology that is disjoint with theannotation concept being tested. The set of concepts tobe covered in this way can be discovered by subtractingfrom the domain designated by a given concept, theinstances that belong to its subconcepts and siblings12

in the ontology. For example, using this method, parti-tioning the domain of Sequence results in the set of disjointsub-domains designated by the following concepts:

• Sequence u ¬(NucleotideSeq t ProteinSeq),• NucleotideSeq u ¬(DNASeq t RNASeq t ProteinSeq),• ProteinSeq u ¬NucletotideSeq ,• DNASeq u ¬RNASeq ,• RNASeq u ¬DNASeq .

5 REAL-WORLD EVALUATION

To assess the effectiveness of the verification methoddescribed in this paper, we run an experiment with theobjective of measuring the defect-detecting power of thetest suites generated using the partitioning strategiesproposed. In doing so, we used as a base line forcomparison, tests conducted using randomly generatedtest cases. Specifically, given semantic annotations ofweb services (which will be presented later), we createda test suite for those annotations using the All-Disjoint-Concepts strategy, and created a randomly generatedtest-suite of the same size. By randomly generated, wemean that the values of the input parameters in the testsuite were selected randomly. Similarly, we created atest suite for verifying semantic annotations using theImmediate-Sub-Super-Concepts strategy, and created arandomly generated test-suite of the same size.

To assess the effectiveness of the partitioning strate-gies, we used the two following measures:

• Increase in Recall: we use this measure to comparethe defect-detecting power of the test suite gen-erated using the partitioning strategies, with thatof a randomly generated test suite of the samesize. Increase in recall is defined as the ratio of thenumber of true positive defects that are discoveredusing the partitioning strategy to the number oftrue positive defects that are detected using the ran-domly generated test suite of the same size. A valueabove 100% means that the test suite generatedusing the partitioning strategy performs better thanthe corresponding randomly generated test suite.

• Precision: False positive errors in annotations canbe detected due to the difficulties in dealing withreal web services, with their varying availability andnon-standard termination routes. We used precision

12. By “siblings” here, we mean two concepts which share animmediate superclass.

to estimate the negative effect that false positive de-fects may have on the verification method proposed.Precision is defined as the ratio of the number oftrue positive errors that are detected to the sumof the numbers of true positive and false positiveerrors. A higher precision means that fewer falsepositives are reported using the verification method.

Experiment Setup

A large number of bioinformatics web services are nowpublicly available, e.g., the myGrid toolkit provides ac-cess to over 3000 third party web services. A domain ex-pert from the myGrid team annotated a number of these,and the results were made available through the Fetaservice registry [14]; 53 accessible web services were an-notated at the time of writing13. These annotations werecreated in order to facilitate service discovery [14] andto guide workflow composition [15], and we thereforeassumed a strong semantics when verifying the semanticannotations of input parameters. Interested readers canfind information about the web service operations andsemantic annotations that were used in the experimentonline14.

We created a pool of annotated instances using prove-nance logs supplied by the myGrid project, and usedthem to create test cases to verify all parameters of theapplicable web services in Feta. In total, we tested 69web service input parameters. In order to examine theeffects of different partitioning strategies on the resultsof the test execution, we created four different test suites,using the following selection strategies from the initialinstance pool:

(i) To assess the effectiveness of the All-Disjoint-Concepts strategy, we selected data from the poolusing this partitioning strategy. This produced 1819test cases.

(ii) As a baseline selection criterion, for comparison, weselected an input data set with the same numberof instances as in case (i) but where the instanceswere randomly selected from the input pool. Thisproduced 1819 test cases.

(iii) To assess the effectiveness of the Immediate-Sub-Superconcepts partitioning strategy, we selecteddata from the pool using this strategy. This pro-duced 165 test cases (a substantial reduction on (i)).

(iv) We selected an input data set with the same numberof instances as selected in case (iii) but where theinstances were randomly selected from the inputpool. This produced 165 test cases.

Test suites were then generated from these selected valuesets, and the test cases executed.

13. The number of web services in the Feta service registry exceeds53; however, many were unsuitable for use in our experiment sincethey either had some un-annotated parameters or else were no longeravailable at the endpoint specified by the WSDL files.

14. img.cs.man.ac.uk/quasar/experiment info.php

11

0"5"

10"15"20"25"30"35"40"45"

i)"All,Disjoint,Concepts"strategy"

ii)"Random:"pool"of"the"same"size"as"(i)"

True"posiEves" False"posiEves"

Fig. 7. Comparison of Defects Detected Using the All-Disjoint-Concepts and Random Strategies

0"5"

10"15"20"25"30"35"40"45"

iii)"Immediate"Sub3Super3Concepts"

strategy"

iv)"Random:"pool"of"the"same"size"as"(iii)"

Fig. 8. Comparison of Defects Detected Using theImmediate-Sub-Superconcepts and Random Strategies

Results

For each test suite generated, we examined the resultsand classified them according to whether they cor-rectly revealed an error (true positives), or incorrectlyrevealed an error (false positives). Figure 7 shows theresults for the test suite generated using the All-Disjoint-Concepts partitioning strategy, and the correspondingrandomly generated test suite of the same size, andFigure 8 shows the results for the test suite generatedusing the Immediate-Sub-Super-Concepts partitioningstrategy, and the corresponding randomly generatedtest suite of the same size. To help analyze such re-sults, Table 1 shows the precision of of the All-Disjoint-Concepts strategy, the precision of the Immediate-Sub-Super-Concepts strategy, as well as the increase in re-call that we recorded in the results of the All-Disjoint-Concepts (resp. Immediate-Sub-Super-Concepts) strat-egy compared with the corresponding random testingstrategy.

The results show that, in each case, random testingis significantly less effective than the more systematicapproach based on ontology-based partitioning strate-

TABLE 1Analysis of Immediate-Sub-Superconcepts Strategy

Results.Strategy Precision Increase

in recallAll-Disjoint-Concepts 81% 220%

Immediate-Sub-Superconcepts 82% 640%

gies (as suggested by the increase in recall see Table 1),thereby providing evidence in favour of our hypothesisregarding the usefulness of the ontology for partitioningthe input domains. Moreover, all errors discovered bythe random selection approaches were also discoveredby the partitioning strategies. The results also show thatthe number of false positives is relatively small com-pared with the number of true positives. Indeed, bothpartitioning strategies yield a precision that is higherthan 80%.

The results also show the effectiveness of Immediate-Sub-Superconcepts partitioning. A total of 32 true er-rors were discovered, a number which compares veryfavourably with the number of errors found by the All-Disjoint-Concepts partitioning strategy, despite the factthat our more selective partitioning strategy generatedless than a tenth of the test cases of the All-Disjoint-Concepts strategy.

These results also show the effectiveness (in this ex-ample) of the Immediate-Sub-Superconcepts partitioningstrategy in finding over-generalisation errors of this sort.This raised the question of whether the strategy wasequally successful at detecting over-specialisation errors,i.e. where the concept used for annotation is a sub-concept of the correct annotation.

Since our collection of annotations did not containmany errors of this kind, we seeded 7 errors, that arerepresentative of over-specialisation, in the annotationsof the inputs. Specifically, we randomly picked 7 inputsparameters that were found not to be erroneous usingthe All-Disjoint-Concepts partitioning method. For eachof those input, we modified its annotation by associatingit with a concept c that is a direct sub-concept of theconcept used for its annotations c. Where the conceptc has more than one sub-concept, we randomly pickedone of its sub-concepts and used it to annotate the inputparameter in question. We then ran the Immediate-Sub-Superconcepts test cases again, and were able to discover5 of the seeded errors. The remaining 2 errors were notlocated because we were unable to find instances for oneof the partitions proposed by this strategy. Specifically,this was the concept: DNASequence u ¬RNASequence. Thisis partly due the fact that the definitions of the conceptsDNASequence and RNASequence in the ontology used forannotation are not complete: no bioinformatics sequencecan be both a DNA sequence and an RNA sequence;however, such a restriction was not specified in the on-tology. As a result, even though the repository contains aDNA sequence, we were not able to use that instance torepresent the partition: DNASequence u ¬RNASequence.

12

This kind of problem, i.e., incomplete definitions ofconcepts, is common in ontology specification [22].

It is worth noting that unlike the All-Disjoint-Conceptsstrategy, the Immediate-Sub-Superconcepts strategy didnot manage to discover errors arising from the accep-tance of instances that do not belong to the annotation.This can be explained by the fact that usually to uncoverthose errors, we need instances of concepts that are notneighbours of the concept used for annotation, but ratherare disjoint concepts that are often not descendants orascendants of the incorrect annotation.

As discussed earlier, to verify the annotations of out-puts, we do not generate test data for the outputs usingontology-based partitioning strategies (since we do nothave an oracle for detecting errors using generated testdata). Instead, we use the data instances delivered bythe outputs as a result of the operation invocationsperformed for testing the annotations of inputs. To testthe annotations of outputs, we located, for each of them,a service operation having an input annotated using thesame concept as the output under test. In the case wherethe service registry did not contain any operation withsuch an input, we located a service operation with aninput annotated using a concept that is disjoint with theannotation of the output parameter under test. Usingthis method, we located 16 operations: 9 having aninput with the same annotation as the output and 7annotated using a concept that is disjoint with the outputannotation. We then used the instances delivered by eachoutput to feed the execution of the service operationlocated for it.

These test case numbers were small, especially whencompared with the large number of test cases we hadbeen able to generate for the inputs. Also, we wereable to discover only one error in the annotation of theoutputs by the execution of this test suite. The output ofthe operation get reaction by enzyme was annotated usingthe concept KeggRecord whereas the instances it deliveredwere accepted by an input that is annotated using Path-wayReactionID, which is disjoint with KeggRecord.

These poor results can be partly explained by thefollowing observation. The instances used for testingthe annotation of an output are not necessarily a goodrepresentation of the domain of values of the output:we did not generate the instances using a partitioningstrategy, but instead relied on the instances delivered as aresult of tests generated with input parameters in mind.

The above observation prompts us to elaborate and as-sess a new strategy for testing the annotations of outputparameters. The proposed strategy exploits the fact thatmany service operations are used within workflows thatproduce provenance traces containing instances valueof operation outputs. Those readily available values areobtained over time as a result of multiple executions ofworkflows, and are more likely to yield sets of valuesthat are better representations of the domain of outputs(compared with the case in which we only rely on theoutput values obtained as a result of testing operation

inputs). Of the 53 service operations under test, 19 wereused in the composition of workflows in the myExper-iment repository15. Those 19 operations had in total 27output parameters. We executed those workflows usingas many legal input values as we could harvest, eitherfrom the workflow descriptions in myExperiment orfrom the authors or users of the workflows. On average,we executed each workflow 24 times using differentlegal input values (we did not use our partitioniningmethod since our objectve is not test the input butthe ouput parameters). We then collected provenancetraces of those workflows, which contained the instancevalues produced by the service operation. To test theannotations of the output of those operations, we usedthe same method as earlier, i.e., by feeding their valuesto service operations the inputs of which have the sameannotation as the operation output subject to test. Usingthis method, we were able to identify 11 additional errorsin the annotation of output parameters. The errors iden-tified resulted from over-specialization in annotations.For example, the curator described the output of theoperation getUNIPROTEntry as a protein sequence,whereas that operation was also able to produce proteinrecords as well as protein sequences.

To summarise, the experimental evaluation reportedin this section showed that:

• Real world semantic annotations suffer from errors.Almost half of tested inputs were found to haveerroneous annotations.

• Partitioning strategies allow a more effective discov-ery of errors in the annotations of inputs comparedwith randomly generated test data. The precision ofof the tests conducted using the partitioning strate-gies is higher than 80%, and their effectiveness interms of recall is much higher than random testing.

• Regarding the annotation of output parameters, theempirical evaluation showed that the use of outputvalues, that are obtained using input values selectedbased on the proposed partitioning strategies, is notan effective means for testing. Rather, other sourcesof information, e.g., the provenance traces producedby workflow executions seem to be more effective.

6 RELATED WORK

In this section, we review proposals that are related toours. In doing so, we structure the section into threesubsections: annotation, verification, and use, representingthe steps of the semantic annotation lifecycle presentedin Section 3.1.

6.1 Web Service AnnotationSeveral tools have been proposed by the semantic webservice community for assisting human curators in theannotation task. For example, Radiant [12] is an Eclipse

15. http://www.myexperiment.org

13

plug-in based tool that provides a graphical user inter-face for annotating web services, by decorating the webservice WSDL document with concepts from domainontologies. To facilitate the annotation task, the authorsof Radiant, developed Meteor-S [20], a tool that semi-automatically annotates web services by suggesting tothe user concepts from domain ontologies that can beused for annotation. At the heart of the tool, there areschema matching and machine learning techniques thatautomatically identify a set of candidate concepts thatcan be used for the annotation of web services. APIHUT[11] is a tool for the annotation, indexing and discoveryof Web APIs, e.g., Rest Web Services. Using APIHUT, thefacets of a web service are automatically determined bycomparing the description and tags associated with theweb services, with the vectors of terms characterizingthe facet’s values using the cosine similarity metric.Built on APIHUT, KINO [21] is a generic tool for an-notating web APIs, including Rest web services. Themain difference between APIHUT and KINO, is thatthe latter allows users to utilize the ontology of theirchoice for annotation. Furthermore, Kino automaticallyidentifies the ontologies (and concepts) that can be usedfor annotation.

We have also shown in a previous proposal [2] howinformation about semantic annotations of web serviceparameters can be inferred based on their connectionsto other (annotated) operation parameters within work-flows. The idea behind this work is that if an inputparameter is connected to an output parameter within aworkflow and that the connection is free from mismatch,then the semantic domain of the output is a sub-conceptof the semantic domain of the input. This allows infer-ring information about the semantic annotation of nonannotated parameters from the semantic annotation ofthe parameters, to which they are connected to withinworkflows.

We regard our work on the verification of semanticannotations of web services, as complementary to theabove proposals. While the above proposals focus onassisting the human user in annotating web services, themethod proposed in this paper can be used as a system-atic means to assess the accuracy of annotations beforethey are deployed for public use. This is illustrated inthe semantic annotation lifecycle in Figure 1.

6.2 Semantic Annotation VerificationTo the best of our knowledge, there are no proposals inthe semantic web service literature for verifying serviceannotations. Instead, semantic annotations are used bya handful of proposals as inputs for specifying the testsuites used for verifying web services’ behaviour [19].That is, semantic annotations are regarded as beingthe correct description of the service’s behaviour, andthe test cases created are intended to reveal defects inthe service. For example, Tsai et al. propose to extendthe WSDL document with semantic annotations that

describe, among other things, the dependencies betweenthe web service inputs and outputs, and to use theseannotations for specifying the test suites used for ver-ifying web services [23]. Other proposals use semanticannotations to describe service composition, and usethese descriptions as input to test the resulting compositeservice [24]. For example, Wang et al. use OWL-S todefine workflows specifying composite services [24]. Totest a composite service, they generate test cases thatexercise all possible paths in its associated workflow.

The above proposals assume that semantic annotationsaccurately specify web services’ behaviour. This is rathera strong assumption that does not always hold in prac-tice, as shown by the experimental results in the previoussection. In practice, a web service is often a black boxannotated after its deployment by third party annotatorsthat have been involved with neither the design nor theimplementation of the service. The method presented inthis paper is therefore meant to help these annotators ininspecting their annotations for errors.

Among the existing tools that assist human users infunctional testing of web services, there is SoapUI [8]. Itis a toolkit that provides the means for specifying andexecuting test suites to verify the functionality of givenweb service operations. Although more targeted towardtesting the functionality of web services, SoapUI can beused together with QuASAR, the tool that implementsthe verification method presented in this paper. QuASARprovides functionalities for i) generating test cases, ii)executing web services using the input specified in thetest cases, and iii) analysing execution results. The sec-ond step (ii) (which corresponds to phases 2 and 3 inthe verification process illustrated in Figure 1) can beperformed by SoapUI, since it is built to support largenumbers of web service calls.

6.3 Uses of Semantic AnnotationsSemantic annotations of web services can be used bytools to assist in service discovery and composition[20], [21], [7], [1]. For example, Metor-S [20] providesa selection service that locates web services based ontheir semantic annotations. Kino [21] also provides themeans for locating web services. In Kino, Web APIs(including web services) are indexed based on theirsemantic annotations. The resulting indexes are thenused to answer user queries.

Semantic annotations of web services are also used toguide the composition of scientific workflows [7], [25].For example, Dhamanaskar et al. developed a servicesuggestion engine [7] that is interfaced with the scientificworkflow platform Galaxy. The service suggestion en-gine guides the composition of workflows by providingthe designer with a list of web services that can beused to complete the workflow being designed. Theweb services are retrieved based on two criteria: i) thefunctionality implemented by the service: the designerindicates the task that the web service should fullfil,

14

and ii) compatibility in the semantic types between theoutput of the preceding web service in the workflow andthe suggested web service. Semantic annotations of webservice parameters are used to check the second condi-tion. Similarly, in previous work [1], we have developeda plugin for the Taverna workflow system, to check thatconnected input and output parameters are compatiblewithin a workflow. In doing, so we made use of semanticannotations of web services.

It is worth mentioning that semantic annotations ofweb service parameters are partial descriptions of webservices in the sense that they do not describe thetransformations carried out by the web service. Yet, theyare crucial in service discovery and guiding workflowcomposition. For example, to guide workflow compo-sition, they are used to suggest web service operationswith parameters that are semantically compatible withthose in the (incomplete) workflow. The list of serviceoperations retrieved can then be examined by the work-flow designer to identify the service operations that aresuitable for the task at hand. Where information aboutthe task carried out by the suggested service operationsis available, then it can be used to filter out those servicesoperations that do not implement the task required bythe workflow designer, e.g., [7].

In summary, the method for verifying semantic anno-tation of web services that we presented in this paperis complementary to existing proposals in the literature.It closes the gap in the web service annotation lifecy-cle, by providing a systematic means for assessing thevalidity of semantic annotations specified using toolssuch as Meteor-S and Radiant, before the annotationsare deployed for public use in web service discoveryand composition.

7 CONCLUSIONS

In this paper, we have described a first step towards pro-viding low-cost tools to assist annotators in verifying thesemantic descriptions they create before they are madepublicly accessible. The approach has been implementedwithin the QuASAR toolkit, and is available as an opensource system16.

Despite the complications of working with real webservices, which do not conform to the standard termi-nation models and which are subject to the exigenciesof a dynamic and unpredictable communications net-work, our approach to checking annotation semantics bymeans of operation acceptance proved to be surprisinglyeffective as far as annotations for input parameters areconcerned. In addition, adequacy criteria based on theuse of the ontology for partitioning demonstrated a clearability to improve the defect-detection power of testssets when compared with randomly selected suites ofthe same size.

16. http://img.cs.manchester.ac.uk/quasar

REFERENCES[1] K. Belhajjame, S. M. Embury, and N. W. Paton. On characterising

and identifying mismatches in scientific workflows. In 3rd DILS06, pages 240–247. Springer, 2006.

[2] K. Belhajjame, S. M. Embury, N. W. Paton, et al.. AutomaticAnnotation of Web Services Based on Workflow Definitions.TWeb, 2(2):1–34, 2008.

[3] M. Brambilla, S. Ceri, F. M. Facca, I. Celino, D. Cerizza, and E. D.Valle. Model-driven design and development of semantic webservice applications. ACM Trans. Internet Techn., 8(1):1–31, 2007.

[4] M. H. Burstein, J. R. Hobbs, O. Lassila, et al.. DAML-S: WebService Description for the Semantic Web. In ISWC, pages 348–363, London, UK, 2002. Springer-Verlag.

[5] J. Cardoso and A. P. Sheth, editors. Semantic Web Services, Processesand Applications. Springer, 2006.

[6] S. B. Davidson and J. Freire. Provenance and scientific workflows:challenges and opportunities. In SIGMOD, ACM, 2008.

[7] A. Dhamanaskar, M. E. Cotterell, J. Zheng, et al. Suggestions forGalaxy Workflow Design Using Semantically Annotated Services.In FOIS, 2012.

[8] EVIWARE. SoapUI; the Web Services Testing tool.http://www.soapui.org/. accessed 2012-11-07.

[9] H. Foster, S. Uchitel, J. Magee, and J. Kramer. Model-basedVerification of Web Services. In ASE, pages 152–161. IEEE, 2003.

[10] A. Gargantini and C. Heitmeyer. Using Model Checking to Gen-erate Tests from Requirements Specification. Software EngineeringNotes, 24(6):146–162, November 1999.

[11] K. Gomadam, A. Ranabahu, M. Nagarajan, et al.. A FacetedClassification Based Approach to Search and Rank Web APIs. InICWS, IEEE, 2008.

[12] K. Gomadam, K. Verma, and D. Brewer, et al.. Radiant: A toolfor semantic annotation of Web Services. In ISWC, (Demo Paper),2005.

[13] Richard G. Hamlet. Theoretical comparison of testing methods.In Symposium on Testing, Analysis, and Verification, pages 28–37.ACM, 1989.

[14] P. W. Lord, P. Alper, C. Wroe, and C. A. Goble. Feta: A light-weight architecture for user oriented semantic service discovery.In ESWC, pages 17–31. Springer, 2005.

[15] P. W. Lord, S. Bechhofer, M. D. Wilkinson, et al.. Applying seman-tic web services to bioinformatics: Experiences gained, lessonslearnt. In ISWC, pages 350–364. Springer, 2004.

[16] E. M. Maximilien and M. P. Singh. A framework and ontology fordynamic web services selection. IEEE Internet Computing, 8(5):84–93, 2004.

[17] B. Medjahed, A. Bouguettaya, and A. K. Elmagarmid. Composingweb services on the semantic web. VLDB Journal, 12(4):333–351,2003.

[18] B. Meyer. Seven Principles of Software Testing. IEEE Computer,41(8): 99-101 (2008)

[19] G. Oghabi, J. Bentahar, and A. Benharref. On the Verification ofBehavioral and Probabilistic Web Services Using Transformation.In ICWS, 548–555, IEEE Computer Society, 2011.

[20] A. A. Patil, S. A. Oundhakar, A. P. Sheth, and K. Verma. Meteor-s web service annotation framework. In WWW, pages 553–562.ACM, 2004.

[21] A. Ranabahu, P. Parikh, M. Panahiazar, et al. Kino: A GenericDocument Management System for Biologists Using SA-RESTand Faceted Search. In ICSC, IEEE, 2011.

[22] A. L. Rector, N. Drummond, M. Horridge, et al.. Owl pizzas: Prac-tical experience of teaching owl-dl: Common errors & commonpatterns. In EKAW, pages 63–81. Springer, 2004.

[23] W. T. Tsai, R. Paul, Y. Wang, C. Fan, and D. Wang. Extendingwsdl to facilitate web services testing. In HASE ’02, page 171.IEEE, 2002.

[24] Y. Wang, X. Bai, Juanzi Li, and R. Huang. Ontology-Based TestCase Generation for Testing Web Services. In ISADS’07. IEEE,2007.

[25] D. Withers, E. A. Kawas, E. L. McCarthy, et al.. Semantically-Guided Workflow Construction in Taverna: The SADI andBioMoby Plug-Ins. In ISoLA (1), pages 301–312, 2010.

[26] C. Wroe, R. Stevens, C. A. Goble, et al.. A suite of daml+oilontologies to describe bioinformatics web services and data. Int.J. Cooperative Inf. Syst., 12(2):197–224, 2003.