-
A User-Centered Methodology for the Evaluation of(Semantic) Web
Service Discovery and Selection
Friederike Klan, Birgitta König-Ries
friederike.klan|[email protected]
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 2
Service Oriented Architectures
service consumer
service provider
semanticmatchmaking
service provider
.
.
.
invocation
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 3
Web Service Discovery and Selection – B2B Scenario
manufacturerinventory managementof a retailer
semanticmatchmaking
manufacturer
.
.
.
invocation
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 4
Web Service Discovery and Selection – B2B Scenario
manufacturerinventory managementof a retailer
semanticmatchmaking
manufacturer
.
.
.
invocation
service requirements known in advance
automatic selection by a machine
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 5
Web Service Discovery and Selection – B2C Scenario
provider
flight bookingweb portal
semanticmatchmaking
provider
.
.
.
invocation
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 6
Web Service Discovery and Selection – B2C Scenario
provider
flight bookingweb portal
semanticmatchmaking
provider
.
.
.
invocation
service requirements NOT known in advance
manual selection by a human
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 7
Changed System Requirements
interactive and incremental requirements elicitation
provide incentives for requirements construction andencourage to
make tradeoffs
R1
R2
service requirements NOT known in advance
manual selection by a human
educate about relevant service alternativesR3
avoid inconsistent selection decisionsR4
maintain accurate requirements modelR5
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 8
Evaluation Methodology
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexhsystem
usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
US
ER
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 9
Evaluation Methodology
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
initialsystem model
SMinil system
usage
SY
STE
MU
SE
R
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 10
Verifying Requirements – Requirements Construction (R2)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
initialsystem model
SMinil system
usage
SY
STE
MU
SE
R
R2
R2
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 11
Verifying Requirements – Service Alternatives (R3)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
initialsystem model
SMinil system
usage
SY
STE
MU
SE
R
R3
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 12
Verifying Requirements – Consistent Selection (R4)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
ranking based
on MMuse
RankingMMuse
initialsystem model
SMinil system
usage
SY
STE
MU
SE
REV
ALU
ATI
ON R4
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 13
Verifying Requirements – Accurate Model (R5)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
ranking based
on SMfinal
RankingSMfinal
selection after exhaustive
search
Selexh
expressed requirements
ranking based
on MMuse
RankingMMuse
initialsystem model
SMinil system
usage
SY
STE
MU
SE
REV
ALU
ATI
ON
R5R5
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 14
Conversational Service Selection
service EFFECT entity from to carrier departure price
service1 Owned Transportation Ticket
Jena Paris Train 12.09.2012 10:38
156EUR
service2 Owned Transportation Ticket
Jena Paris RailAndFly 12.09.2012 10:50
571EUR
service4 Owned Transportation Ticket
Jena Paris RailAndFly 12.09.2012 11:50
571EUR
service5 Owned Transportation Ticket
Jena Paris RailAndFly 12.09.2012 12:20
service6 Owned Transportation Ticket
Jena Paris Train 12.09.2012 12:38
156EUR
...
less
Available attributes ...
trip duration (100%)
intermediate hops (82%)
airline (49%)
Available subtypes ...carrier Train (46%)
RailAndFly (49%) Bus (5%)
Tradeoff opportunities ...trip duration , # intermediate hops
>1 (82%)
airline!=Lufthansa (100%)trip duration , departure (70%)
91% of the offers fulfill your requirements.
add serviceattributes
refine serviceattributes
compromiserequirements
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 15
Case Study – Evaluation Setting
study participants
10 test users (6 males, 4 females, age 25 − 58 years) 9
experience with online purchasing 2 of them familiar with Web
Services
service offers
generated from structured information about computer items from
8 categories (desktop PC, notebook, e-book reader, …)
participants had to choose from 200 services of one category
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 16
Case Study – Assessing Service Requirements
asked participants to indicate service aspects that are
important to them specify their requirements on these aspects and
to weight the indicated service aspects against each other
put no restrictions on type of these requirements or the way of
specifying them
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 17
Selected Results – Requirements Construction (R2)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
initialsystem model
SMinil system
usage
SY
STE
MU
SE
R
R2
R2
respondents added/revisedand abandoned requirements
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 18
Selected Results – Requirements Construction (R2)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
initialsystem model
SMinil system
usage
SY
STE
MU
SE
R
R2
R2
no hidden requirements
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 19
Selected Results – Service Alternatives (R3)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
initialsystem model
SMinil system
usage
SY
STE
MU
SE
R
R3
just one switch(due to better alternative)
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 20
Selected Results – Consistent Selection (R4)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
selection after exhaustive
search
Selexh
expressed requirements
ranking based
on MMuse
RankingMMuse
initialsystem model
SMinil system
usage
SY
STE
MU
SE
REV
ALU
ATI
ON R4
mean rank was8.33 ±5.33 (out of 200)
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 21
Selected Results – Accurate Model (R5)
initialmental model
MMinit
mental modelafter system
usage
MMuse
mental modelafter exhaustive
search
MMexh
finalsystem model
SMfinal
system usage browsing
selectionafter system
usage
Seluse
ranking based
on SMfinal
RankingSMfinal
selection after exhaustive
search
Selexh
expressed requirements
ranking based
on MMuse
RankingMMuse
initialsystem model
SMinil system
usage
SY
STE
MU
SE
REV
ALU
ATI
ON
R5
● mean rank difference of the top ten offers was 30.98 ±
8.50
● 23.04 ± 7.88 for the 5 fairly consistent weightings
-
2014/06/02 Friederike Klan: A User-Centered Evaluation
Methodology for SWS Selection 22
Summary
identified requirements to service selection in B2C
scenarios
presented a user-centered evaluation methodology w.r.t. these
requirements
case study demonstrated feasibility and appropriateness of the
suggested approach
