Coercing Clients into Facilitating Failover for Object Delivery

Wyatt Lloyd, Michael J. FreedmanPrinceton UniversityTRODS Transparent Recovery forObject Delivery Services

Presented at DSN-DCCS 2011 in Hong Kong on 6/28/111

223

34

4ClientServerServerServiceServerServerConnectionRecovered!55Object Delivery ServicesRead-Only

Static Content

Webpages, Images, Videos6

6Work NowCant Modify Clients7

7Key IdeaCoerce client to helpTo identify connections that need recoveryTo reliably store information

Yet client is unmodified and unawareExploit TCP spec to control clients stack88Object Delivery Cluster9ServerServerServiceServerServerLiveness MonitorLoad Balancer9Failure10ServerServerServiceServerServerLiveness MonitorLoad Balancer10TRODS11ClientServerServiceServerServerLiveness MonitorLoad Balancer?11TRODS12ClientServerServiceServerServerLiveness MonitorLoad Balancer?Store12Road to RecoveryStepTechniqueRedirect to live server.Liveness monitor updates load balancer

Induce client to send packetCoerce clients TCP stack

Continue ConnectionDetermine PhaseUse packet + stored infoIdentify Object.Stored InfoFind Offset..Use packet + stored info

13NewNew13Coercing ClientsAlways Leave A Packet UnacknowledgedSYNSYN/ACKACKRequestResponse1ACKResponse2ACKResponse3FIN/ACKFINClientServer14SYNRequestRetransmit QueueRetransmit QueueSYN/ACKResponse1Response2Response3FIN/ACKFINACKACKAlways Something HereExploit TCP Spec for Recovery Initiation!14Continuing the ConnectionDetermine Phase:TCP SetupHTTP SetupHTTP DownloadTCP Teardown15TRODS Saves Info15Continuing the DownloadHTTP ObjectID

Offset16= TCP Ack HTTP ObjectISN

HTTP ObjectISNTCP AckTCP ISNHTTPRespHeaderHTTPObjectSYN16Continuing the DownloadHTTP ObjectID

Offset17= TCP Ack HTTP ObjectISN

HTTP ObjectISNTCP AckTCP ISNHTTPRespHeaderHTTPObjectSYN17Persistent StoreKey-Value Store+ Corner Cases Handled+ Unlimited ObjectsStill Efficient (1 save only)

TCP Timestamp+ Very Efficient (1 machine only)1 Million Object LimitCorner Cases

1818IPTCPPayloadTSKVExploit TCP Spec for Persistence!18Recover the ConnectionInitiate New ConnectionGET ObjectID Range: bytes=Offset-

Splice Connections Together

Works with Unmodified Servers!19TRODS20Packet ManipulationServerTCPIPTRODS

IPTCPIPTCP20ServerTCPIPTRODS

TRODS21Packet ManipulationProtocol InspectionRequestRequestObjISNObjIDResponse121ServerTCPIPTRODS

TRODS22Packet ManipulationProtocol InspectionBlocks ConnectionResponse1ObjISNObjID22ServerTCPIPTRODS

TRODS23Packet ManipulationProtocol InspectionBlocks ConnectionState InjectionIPTCPIPTCPTS23TRODS24Packet ManipulationProtocol InspectionBlocks ConnectionState InjectionRecovery InitiationServerTCPIPTRODS

Ack?24Server

TCPIPTRODSServer

TCPIPTRODSServer

TCPIPTRODSFailure Walkthrough2525ClientServiceLiveness MonitorLoad BalancerKV StoreISNIDSYNACKRequestSYN/ACKResponse125Server

TCPIPTRODSServer

TCPIPTRODSFailure Walkthrough2626ClientLiveness MonitorLoad BalancerKV StoreACKResponse2?!ACKResponse3ACKResponse4FINFINACKISNIDService26Related WorkNew TransportTrickles, SCTP, TCP Migrate,

TCPFT-TCP, ST-TCP, Backdoors,

HTTPCoRAL, 2727ImplementationLinux Kernel Module

3,000 lines of C

~CoRALOptimistic subset of CoRAL2828ExperimentsAdditional LatencyNormalFailure

ThroughputLighttpd @ PrincetonApache @ EmulabHybrid TS & KV ThroughputFailure29Normal Case LatencyTRODS-TimeStamp (TS)Median: + 0.009 ms99th: + 0.012 ms

TRODS-Key-Value (KV)Median: + 0.137 ms99th: + 0.148 ms3030

Recovery Latency31Blink of an eye~35%~50%~15%31ThroughPut Per Server32120 ops/s120 ops/sRaw30 ops/s30 ops/sFrontend30 ops/s/server20 ops/s/serverTPPS32

33

KV/Server: 1/8KV/Server: 1/34KV/Server: 1/4KV/Server: 1/2Lighttpd9%38%

7%66%3334Apache

34SummaryRecover Object Delivery Connections

Exploit TCP Specification to Coerce ClientsTo send recovery-starting packetsTo provide persistent storage

EvaluationLow LatencyHigh Throughput Per Server35Unmodified^Unmodified^35SummaryRecover Object Delivery Connections

Exploit TCP Specification to Coerce ClientsTo send recovery-starting packetsTo provide persistent storage

EvaluationLow LatencyHigh Throughput Per Server36Questions?Unmodified^36