HTTP-Tester- A web HTTP-Tester- A web appliance testing toolappliance testing tool

Kiran KomaravoluKiran Komaravolu

Dept of Computer Science and Dept of Computer Science and EnggEngg

Lehigh UniversityLehigh University

What is the need to test What is the need to test Web AppliancesWeb Appliances

Need to use best components.Need to use best components. Compare performance of different web Compare performance of different web

devices.devices. Verify if a device is performs as per Verify if a device is performs as per

specifications.specifications. Choosing between different service Choosing between different service

providers.providers.

Related WorkRelated Work

The Ethereal network analyser (The Ethereal network analyser (www.ethereal.comwww.ethereal.com).).

TCPDUMP/PCAP (TCPDUMP/PCAP (www.tcpdump.orgwww.tcpdump.org).). Httpflow (Grevnin and Davison)Httpflow (Grevnin and Davison) Simultaneous proxy evaluation. Simultaneous proxy evaluation.

(Davison).(Davison). Medusa Proxy. (Koletsou and Medusa Proxy. (Koletsou and

Voelker).Voelker).

Http-TesterHttp-Tester

Replays captured requests to an Replays captured requests to an existing server to one that is being existing server to one that is being evaluated.evaluated.

No active deployment needed.No active deployment needed. Does not affect existing services.Does not affect existing services. Can be used to evaluate any Web Can be used to evaluate any Web

appliance running HTTP.appliance running HTTP.

HTTP-TesterHTTP-Tester Topology setupTopology setup

HTTP-TesterHTTP-Tester

ComponentsComponents The Sniffer.The Sniffer.

Logs all packets onto a packet database.Logs all packets onto a packet database. Request Replicator.Request Replicator.

Replays all requests to the “tested device”Replays all requests to the “tested device” The Post Processor.The Post Processor.

Reads the packet logs, extracts HTTP Reads the packet logs, extracts HTTP requests and responses and correlates requests and responses and correlates them.them.

HTTPHTTP

HTTP Message StructureHTTP Message Structure A command line. (GET, POST, OK etc.)A command line. (GET, POST, OK etc.)

GET http://GET http://www.google.comwww.google.com / HTTP 1.0 / HTTP 1.0 Zero or more header lines. Zero or more header lines.

Content-Type: text/html; charset=iso-8859-Content-Type: text/html; charset=iso-8859-11

Control Return + Line Feed.Control Return + Line Feed. Optional Message BodyOptional Message Body

Response messageResponse message

HTTP Transactions. HTTP Transactions. HTTP 1.0HTTP 1.0

No persistent connections.No persistent connections. A new connection is opened up for A new connection is opened up for

every request made, and is closed down every request made, and is closed down after response sent.after response sent.

HTTP 1.1HTTP 1.1 Supports Persistent Connections.Supports Persistent Connections. Client may make a series of requests on Client may make a series of requests on

a connection. Server returns responses a connection. Server returns responses in same order.in same order.

Supports chunk encoding. Supports chunk encoding.

The SnifferThe Sniffer

Blindly captures all relevant network Blindly captures all relevant network packets and logs them.packets and logs them.

The pcap packet capture library has The pcap packet capture library has been used to read packets off the wire.been used to read packets off the wire.

Berkeley DB package has been used to Berkeley DB package has been used to log the packets into a ‘db’ database.log the packets into a ‘db’ database.

The job of efficient memory and disk usage has The job of efficient memory and disk usage has been delegated to DB.been delegated to DB.

Request ReplicatorRequest Replicator

Replays all requests flowing into original Replays all requests flowing into original device onto the tested device.device onto the tested device.

Works in parallel with the sniffer module.Works in parallel with the sniffer module. Looks for TCP SYN packets and Looks for TCP SYN packets and

determines the beginning of a request determines the beginning of a request flow. And opens up a new connection to flow. And opens up a new connection to the second device.the second device.

The payload every next packet sent to on The payload every next packet sent to on this connection is sent to the second this connection is sent to the second device.device.

Request Replicator.Request Replicator.

This way the second server sees the This way the second server sees the same workload as the first one.same workload as the first one.

The replies sent by the server are The replies sent by the server are ignored. They are flushed out as the ignored. They are flushed out as the socket buffers overflow. The sniffer socket buffers overflow. The sniffer module though logs the TCP module though logs the TCP packets.packets.

Post Processing.Post Processing.

Extracts HTTP requests/responses Extracts HTTP requests/responses from the packet logs.from the packet logs.

Every request is mapped onto its Every request is mapped onto its response. The same request response. The same request response pair on the second response pair on the second connection is also identified.connection is also identified.

Httpflow routines used to extract Httpflow routines used to extract HTTP requests from packet stream.HTTP requests from packet stream.

Identifying Requests.Identifying Requests.

We know which way requests flow.We know which way requests flow. Assumption: First TCP packet of Assumption: First TCP packet of

every request has one of every request has one of GET, HEAD,OPTIONS, DELETE, POST, PUT.

End of headers is determined by a End of headers is determined by a CRLF.CRLF.

content-length field is specified in content-length field is specified in case of PUT/POST requests.case of PUT/POST requests.

Identifying responsesIdentifying responses

Response headers extrated by looking Response headers extrated by looking for a CRLF.for a CRLF.

End of Response message determined End of Response message determined in three ways:in three ways: Server sends a FIN or RST packet.Server sends a FIN or RST packet. Response headers contain a “content-Response headers contain a “content-

length” field.length” field. Response is chunk encoded. Each chunk Response is chunk encoded. Each chunk

has its own length. Zero length chunk has its own length. Zero length chunk indicates end of stream.indicates end of stream.

Correlating requests and Correlating requests and responsesresponses

HTTP 1.0HTTP 1.0