An Evaluation of Contemporary Commercial

SOAP Implementations

Presented by: Alex Ng

Alex NgDepartment of Computing,

Macquarie University

Shiping Chen CSIRO ICT Centre

Paul Greenfield CSIRO ICT Centre

AWSA 2004, Melbourne, 13-14 April 2004

Outline Background Experimental design Result discussion Related work Future work

Web Services Definition (W3C)A Web service is a software system

designed to support interoperable machine-to-machine interaction over a network. It has an interface described in a machine-processable format (specifically WSDL). Other systems interact with the Web service in a manner prescribed by its description using SOAP-messages, typically conveyed using HTTP with an XML serialisation in conjunction with other Web-related standards. W3C Web Services Architecture Working Draft (8 August

2003)

W3C Web Services Conceptual Stack

HTTP,HTTPS,SMTP

XML,SOAP,WS-Addressing

XSD,WSDL,UDDI,Policy

Security Reliable Messaging

Transaction

Composition Services Assurances

Description

Messaging

Transport

BPEL4WS

Service Composition

Source: Ferguson D. Secure, Reliable, Transacted Web Services: Architecture and Composition. MSDN technical article, Sept. 2003.

SOAP (Simple Object Access Protocol)

Simple Extensible Based on XML De facto

standard for XML Messaging in supporting Web Services

V1.1(May/00) V1.2 (June/03)

Request POST /StockQuote HTTP/1.1Host: www.stockquoteserver.comContent-Type: text/xml Content-Length: nnnnSOAPAction: “Some-URI”<SOAP:Envelopexmlns:SOAP="urn:schemas.xmlsoap.org:soap.v1"><SOAP:Header>

<t:Transaction xmlns:t=”URI”mustUnderstand=”1”>5</

t:Transaction></SOAP:Header><SOAP:Body>

<m:GetLastTradePrice xmlns:m="URI">

<symbol>DIS</symbol></m:GetLastTradePrice>

</SOAP:Body></SOAP:Envelope>

Other Competitors of SOAP Web Services

Use different transport mechanism Microsoft DIME (Direct Internet Message

Encapsulation) Work with or without SOAP & HTTP send attachments of various types

BEEP (Blocks Extensible Exchange Protocol) Connection oriented, min 257 channels,

asynchronous message exchange Work with or without HTTP

Use different Web Architecture REST (Representational State Transfer)

An architectural style (Roy Fielding) Use XML/HTML, no need for SOAP

Performance Issues of SOAP Web Services

Serialisation and de-serialisation Message size ( 6-8 times larger than

binary) Introduce latency in transmission, storing or

retrieving XML data

XML Processing Parsing (encode/decode XML elements) Validation (conformation to a pre-defined schema) Transformation (XML <-> other format)

Network settings Issues HTTP 1.1 chunking (avoids content-

length calculation) HTTP 1.1 Keep-Alive (reduces connection

set up cost) HTTP 1.1 Pipelining (handles multiple

requests/connection) TCP-Delayed-ACK (defers receiver acknowledgement)

Nagle Algorithm (defers the sending of small packets)

Evaluation of contemporary commercial SOAP implementations

(1) What level of performance is shown by major commercial SOAP Web Services implementations?

(2) How does the performance of SOAP compare to that offered by conventional non-SOAP technologies?

Document/Literal RPC/Literal

Document/Encoded

RPC/Encoded

SOAP Body

Document RPC

SOAP Parameter

Literal

Encoded

Experimental Design

Test Driver

Platform Specific Runner

Property.xml

Test Document

Result Log

Target Application

Server

Target Web Service

100Mbps Ethernet

4x1.6Ghz, 3.6Gb

4x1.6Ghz, 3.6Gb

Invoice: One Customer, 50 product lines

Complex Message

20 customers InquiryMedium Message

Single customer inquiry/update

Simple Message

Implementation Doc/ Lit

Doc/ Enc

RPC/ Lit

RPC/ Enc

Product A

Product B

Product C

 Encoding

StyleSimple Messag

e

Medium Messag

e

Complex

Message

Product A

Doc/Lit 873 7581 21392

Doc/Enc 1593 13973 38842

RPC/Enc 1542 13969 38838

Product BDoc/ Lit 870 7513 21566

RPC/Enc 1597 14035 38999

Product C RPC/Enc 1664 14566 40273

No.of Char.

Result: Message Size

Result: Latency

Result: Simple Message Throughput

0

200

400

600

800

1000

1200

1400

4 16 32 64 128

Concurrent Threads

MP

S

Prd-A Doc/ Enc Prd-A Doc/ Lit Prd-A RPC/ Enc Prd-C RPC/ Enc

Prd-B Doc/ Lit Prd-B RPC/ Enc Prd-R

Result: Medium Message Throughput

0

100

200

300

400

500

600

4 16 32 64 128

Concurrent Threads

MP

S

Result: Complex Message Throughput

0

50

100

150

200

250

300

350

4 16 32 64 128

Concurrent Threads

MP

S

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Document/Literal RPC/Encoded

mS

Client Deserialisation

Network receive

Server Serialisation

Server Application

Server Deserialisation

Network send

Client Serialisation

Result: Serialisation/De-serialisation

23%

13%

30%10%

Evaluation Summary

SOAP Performance is affected by: Encoding style (Document/Literal encoding is the

choice) Product of implementation (a sizable gap between two

products using the same document/literal encoding) Performance delivered by the majority SOAP

implementations: Simple Message – able to deliver reasonably good performance (4

to 6ms vs 1ms non-SOAP) Medium Message – (10 to 22ms vs 7ms non-SOAP) Complex Message – (20 to 63ms vs 15ms non-SOAP)

De-serialisation overhead is higher than that of serialisation.

Related Work (1) The Extreme Lab, Indiana University

Govindaraju, et. al. sending different size of linked list of doubles using

different implementations: SOAPRMI, SUNRMI, and NEXUSRMI, and TCP.

TCP is two orders of magnitude better than SOAPRMI Serialisation and de-serialisation are the bottlenecks.

Chiu, et.al. sending different arrays of mathematical doubles. Suggested Optimization: Pull Parser, schema-specific

XML parser, trie and HTTP 1.1. Kohlhoff and Steele: Compares SOAP against FIX

and CDR SOAP latency is 2 to 3 times worse than CDR FIX is a text based encoding and outperforms CDR and

CDR

Related Work (2) Dhawan: Compares the performance of .NET

Remoting against ASP.NET ASP.NET XML serialisation is more efficient than .NET

Remoting SOAP serialisation

Davis and Parashar Different SOAP Implementations: SOAPRMI/Java1.1,

Microsoft SOAP Toolkit SP2 in VB, Perl SOAP::Lite, Apache SOAP 2.2, and Apache Axis Alpha 3.

inappropriate TCP options had caused 200ms delays in some implementations (MS SOAP, Apache SOAP and SOAP:Lite )

Conclusion: SOAP is in the orders of magnitude slower than JavaRMI and CORBA

Related Work (3) Benkner, et. al.: Compares the performance

of JAXM implementations against JAXRPC in different package sizes (1x1024 -> 8x128) Different Implementations: Apache Axis 1.1

RC2, Sun JWSDP 1.1, Mind Electric GLUE4.0, Systinet WASP 4.5, Novell jBroker Web 2.0, and X-Treme XSOAP 1.2.20.

Conclusion: JAXM & JAXRPC produce similar results Best performance result is achieved when using

large packages of data, i.e. splitting 1MB of source data into four packages of 256KB each for JAXM in Systinet WASP implementation and 2x 512 for JAXRPC in Apache implementation.

Future Work Extend Evaluation to other SOAP

Implementations, such as Apache Axis

Identify other factors when SOAP over HTTP is used across wide-area networks

Your Feedback

Reference Chiu K., et. al. (2002) Investigating the Limits of SOAP Performance for

Scientific Computing In Proceedings of 11th. IEEE International Symposium on High Performance Distributed Computing HPDC-11 2002 (HPDC'02)

Benkner, S., Brandic, I., Dimitrov, A., et al. Performance of Java Web Services Implementations. In Proceedings of ICWS'03. Las Vegas, 2003

Davis, D. and Parashar, M. Latency Performance of SOAP Implementations. In Proceedings of IEEE Cluster Computing and the GRID 2002 (CCGRID'02)

Dhawan, P. Performance Comparison: Exposing Existing Code as a Web Service, October 2001 (on-line) Accessed Date: 10 October 2002http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/bdadotnetarch11.asp

Kohlhoff, C. and Steele, R. (2003) Evaluating SOAP for High Performance Business Applications: Real–Time Trading Systems. In Proceedings of WWW2003. Budapest, Hungary.

Govindaraju, M., A. Slominski, et al. Requirements for and Evaluation of RMI Protocols for Scientific Computing. Supercomputing. IEEE, 2000