IntroductionOur Contribution
Experiments and Results
A Combined LIFO-Priority Scheme forOverload Control of E-commerce Web Servers
Naresh Singhmar Vipul Mathur Varsha ApteD. Manjunath
Indian Institute of Technology - BombayPowai, Mumbai, 400 076, India
International Infrastructure Survivability Workshop, 2004
1 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload at E-commerce Web Sites
UK E-tailers ‘lose £300m’ in Xmassales
“Top UK E-tailers are estimated to have
lost more than £300 million over the
busy Christmas shopping period
because of flaky website performance.”
–The Register, January 15, 2004
E-tail sites failing the Xmas test
“Empirix monitored the websites of 10
of the UKs biggest and best retailers
and found many were failing to take all
the hassle out of Christmas shopping.”
–silicon.com, December 19 2003
Online retail sites strain under‘Black Friday’
“Online retailers failed to complete 1 in
5 transactions during peak hours of the
biggest shopping day of Xmas season”
–InternetWeek.com, December 4, 2003
Iraq conflict hits Web sites hard
“. . . traffic to the site has already almost
tripled and is expected to grow further.
. . . the top 15 news sites have seen
traffic jump by more than 40%.”– BBC
News Online, March 20, 2003
2 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload at E-commerce Web Sites
UK E-tailers ‘lose £300m’ in Xmassales
“Top UK E-tailers are estimated to have
lost more than £300 million over the
busy Christmas shopping period
because of flaky website performance.”
–The Register, January 15, 2004
E-tail sites failing the Xmas test
“Empirix monitored the websites of 10
of the UKs biggest and best retailers
and found many were failing to take all
the hassle out of Christmas shopping.”
–silicon.com, December 19 2003
Online retail sites strain under‘Black Friday’
“Online retailers failed to complete 1 in
5 transactions during peak hours of the
biggest shopping day of Xmas season”
–InternetWeek.com, December 4, 2003
Iraq conflict hits Web sites hard
“. . . traffic to the site has already almost
tripled and is expected to grow further.
. . . the top 15 news sites have seen
traffic jump by more than 40%.”– BBC
News Online, March 20, 2003
2 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload at E-commerce Web Sites
UK E-tailers ‘lose £300m’ in Xmassales
“Top UK E-tailers are estimated to have
lost more than £300 million over the
busy Christmas shopping period
because of flaky website performance.”
–The Register, January 15, 2004
E-tail sites failing the Xmas test
“Empirix monitored the websites of 10
of the UKs biggest and best retailers
and found many were failing to take all
the hassle out of Christmas shopping.”
–silicon.com, December 19 2003
Online retail sites strain under‘Black Friday’
“Online retailers failed to complete 1 in
5 transactions during peak hours of the
biggest shopping day of Xmas season”
–InternetWeek.com, December 4, 2003
Iraq conflict hits Web sites hard
“. . . traffic to the site has already almost
tripled and is expected to grow further.
. . . the top 15 news sites have seen
traffic jump by more than 40%.”– BBC
News Online, March 20, 2003
2 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload at E-commerce Web Sites
UK E-tailers ‘lose £300m’ in Xmassales
“Top UK E-tailers are estimated to have
lost more than £300 million over the
busy Christmas shopping period
because of flaky website performance.”
–The Register, January 15, 2004
E-tail sites failing the Xmas test
“Empirix monitored the websites of 10
of the UKs biggest and best retailers
and found many were failing to take all
the hassle out of Christmas shopping.”
–silicon.com, December 19 2003
Online retail sites strain under‘Black Friday’
“Online retailers failed to complete 1 in
5 transactions during peak hours of the
biggest shopping day of Xmas season”
–InternetWeek.com, December 4, 2003
Iraq conflict hits Web sites hard
“. . . traffic to the site has already almost
tripled and is expected to grow further.
. . . the top 15 news sites have seen
traffic jump by more than 40%.”– BBC
News Online, March 20, 2003
2 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload and its Effects
Overload: Offered load > system capacity
Cause of Overload: sales, big shopping days, serverfailures, breaking news
0
0.5
1
1.5
2
2.5
3
3.5
0 1 2 3 4 5 6 7
Thr
ough
put i
n re
q/se
c
Load in req/sec
Server Throughput
Throughput vs. Load
10
20
30
40
50
60
70
80
90
100
0 1 2 3 4 5 6 7
CP
U U
tiliz
atio
n
Load in req/sec
CPU Utilization
CPU Utilization vs. Load
3 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload and its Effects
Effects of OverloadIncreased response time
Abandonment due to timeouts
Retries ⇒ increase in load
Dramatically deteriorated throughput
E-commerce Web sites lose revenue
Customer experience deteriorates at times of peak usage
Objective of Overload Control
Reduce the amount of lost requests and increase throughput
4 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload and its Effects
Effects of OverloadIncreased response time
Abandonment due to timeouts
Retries ⇒ increase in load
Dramatically deteriorated throughput
E-commerce Web sites lose revenue
Customer experience deteriorates at times of peak usage
Objective of Overload Control
Reduce the amount of lost requests and increase throughput
4 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload and its Effects
Effects of OverloadIncreased response time
Abandonment due to timeouts
Retries ⇒ increase in load
Dramatically deteriorated throughput
E-commerce Web sites lose revenue
Customer experience deteriorates at times of peak usage
Objective of Overload Control
Reduce the amount of lost requests and increase throughput
4 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload Control
Previous WorkFocusses mainly on sophesticated techniques which may bedifficult to implement, or are too generic to be effective forE-commerce Web-servers with dynamic content
Our Work
Focus on simplicity, ease of implementation,and on E-commerce Web-servers
5 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Overload Control
Previous WorkFocusses mainly on sophesticated techniques which may bedifficult to implement, or are too generic to be effective forE-commerce Web-servers with dynamic content
Our Work
Focus on simplicity, ease of implementation,and on E-commerce Web-servers
5 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
Online Retail StorePossible Activities on an On-line Store (Screen shots courtesy Amazon.com)
Main, Browse, Search, Details, Login, Shipping, Payment, Confirm
6 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overload ControlE-commerce Workload
E-commerce Workload Model
EXIT
Transaction StagesBrowsing Stages
$
Details
Search
Browse
Main Page
Login
Shipping
Payment
Confirm $
Most users go only through Browsing stages
Very few proceed to revenue generating Transaction stages
7 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
Proposed Scheme and Architecture
8 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
Key Ideas of Proposed Solution
Increase completion rate of revenue generating requests
Separate queues for each type of request
Transaction queues have strictly higher priority thanbrowsing queues
Relative priority within transaction and browsing based on“utility” of the queue
Increase the overall throughput of Web-server during overload
Using LIFO for browsing queues during overload
Switch between LIFO and FIFO based on thresholds
Always FIFO for transaction queues
9 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
Key Ideas of Proposed Solution
Increase completion rate of revenue generating requests
Separate queues for each type of request
Transaction queues have strictly higher priority thanbrowsing queues
Relative priority within transaction and browsing based on“utility” of the queue
Increase the overall throughput of Web-server during overload
Using LIFO for browsing queues during overload
Switch between LIFO and FIFO based on thresholds
Always FIFO for transaction queues
9 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
E-commerce Workload ModelRepresented as a Markov Chain
EXIT
Browsing Stages Transaction Stages
Details
BrowseLogin
Shipping
Payment
Confirm
Search
Main Page
0.4
0.5 0.5
0.3
0.1
0.9
0.9
0.9
Probability of generating revenue can be used as ‘utility’ value
10 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
Proposed Web-server ArchitectureA prototype Web-server with this architecture has been implemented
ChangeServer Policy
Read ControlParameters
Policy Controller
URLClassifier
URLClassifier
URLClassifier
Worker Thread
Worker Thread
Worker Thread
Queues
Listener
Check Server Policy
S
HEDULER
C
Read RequestRequestProcess
Send Response to Client
11 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
LIFO-Pri Scheme
Set Service Discipline of Browsing Queues
1 Measure CPU Utilization over an interval2 If utilization is more than upper threshold, then set
browsing queue discipline to LIFO3 If utilization is less than lower threshold, then set browsing
queue discipline to FIFO
12 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
LIFO-Pri Scheme
Dynamic Priority
1 When a worker thread is available and at leastone queue has a pending request,
2 Calculate dynamic priority of each queue= queue length × utility
3 Select the queue with highest dynamic priority4 Read a request from this queue according to
current service discipline5 Assign worker thread to request.
13 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
LIFO vs. FIFO: Response TimeResponse time distribution at ρ = 0.941 with a timeout of 20 seconds.
0.001
0.01
0.1
1
0 2 4 6 8 10 12 14 16
P(r
espo
nse
time
> t)
(lo
g sc
ale)
Response time in seconds (t)
Always LIFOAlways FIFO
LIFO at overload
LIFO always has longer tail in non-overload conditions
14 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
LIFO vs. FIFO: Response TimeResponse time distribution at ρ = 1.47 with a timeout of 20 seconds.
0.01
0.1
1
0 5 10 15 20
P(r
espo
nse
time
> t)
(lo
g sc
ale)
Responce time in sec (t)
Always LIFOAlways FIFO
LIFO at overload
P[RLIFO > 15] = 0.1 whereas P[RFIFO > 15] = 0.95
15 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
LIFO vs. FIFO: Response TimeResponse time distribution at ρ = 1.47 with a timeout of 40 seconds.
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30 35 40
P(r
espo
nse
time
> t)
Response time in seconds (t)
Always LIFOAlways FIFO
LIFO at overload
For longer timeout, long tail of LIFO is seen again
16 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
LIFO vs. FIFO: Throughput
Timeout of 40 seconds (ρ = 1.47)
Percentage Always-FIFO Always-LIFO LIFO-at-overloadCompleted 86.7 84.4 84.6Timed-out 00.0 02.3 02.0Dropped 13.3 13.4 13.4
Timeout of 20 seconds (ρ = 1.47)
Completed 21.9 81.0 76.8Timed-out 64.9 05.4 09.7Dropped 13.3 13.6 13.4
Large rate of abandonment in FIFO with a shorter timeout
17 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and ResultsProposed Scheme and Architecture
Observations
Summary of Observations for LIFO vs. FIFO
Longer tail for LIFO ⇒ using LIFO not appropriatewhen offered load < capacity
Larger timeout value favors FIFO (no long tail)
Success rate is higher for LIFO policies in overload(with small timeouts)
LIFO-at-overload gives higher throughput andbetter response time distribution in overload
18 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Experiments and Results
19 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Experimental Setup
Emulate an E-commerce Web site
Eight stages represented by Perl CGI scripts
Modified version of httperf for workload generation
Exponentially distributed timeouts
Retries for requests abandoned due to timeouts
Session abandonments
Separate priority queues for each type of request:4 browsing, 4 transaction
20 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Experiments PerformedThree sets of experiments were done.
Single Queue: FIFO order. Capacity: 100.
8Q Always FIFO: All 8 queues always in FIFO order.Capacity: 50 for browsing queues, 25 for transactionqueues.
8Q LIFO-Pri: LIFO at overload for browsing queues.Always FIFO for transaction queues.
Dynamic priority is used for multi-queue setups. Utility of aqueue is assigned in proportion to probability of a request inthat queue resulting in a final ’confirm’ transaction.
21 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overall Throughput vs. Offered Load
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5
6
0 0.5 1 1.5 2 2.5
Thr
ough
put (
requ
ests
/sec
ond)
Offered Load (normalized)
Single Queue FIFO8Q Always FIFO
8Q LIFO-pri
22 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Average Response Time vs. Offered Load
0
5000
10000
15000
20000
25000
0 0.5 1 1.5 2 2.5
Res
pons
e T
ime
(mill
isec
onds
)
Offered Load (normalized)
Single Queue FIFO8Q Always FIFO
8Q LIFO-pri
23 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Looking at Request TypesThroughput data for different types of requests at ρ = 1.4
Case Requests Browsing Tr-1 Tr-2 Tr-3 Tr-4Generated 42029Completed 16170 20 15 9 8
SQ Timed out 20029 18 5 1 1Dropped 5753Generated 43324 24 20 19 15Completed 19852 23 19 19 15
8Q-AF Timed out 16305 1 1 0 0Dropped 7167 0 0 0 0Generated 44826 195 137 99 53Completed 30851 187 127 87 50
8Q-LIFO-Pri Timed out 4075 8 10 12 3Dropped 9900 0 0 0 0
24 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Looking at Request Types
Requests Completed at ρ = 1.4
Case Browsing Tr-1 Tr-2 Tr-3 Tr-4SQ 16170 20 15 9 88Q-AF 19852 23 19 19 158Q-LIFO-Pri 30851 187 127 87 50
6-7 fold increase in ‘confirm’ requests from SQ to 8Q-LIFO-Pri
25 Singhmar et al Overload Control of E-commerce Web Servers
IntroductionOur Contribution
Experiments and Results
Overall Throughput Data
At ρ = 1.4 (percentages)
Case SQ 8Q-AF 8Q-LIFO-PriCompleted 29.9 36.6 57.5Timed out 36.8 29.9 07.5Dropped 10.6 13.1 18.2Not Generated 22.8 20.4 16.8
26 Singhmar et al Overload Control of E-commerce Web Servers
LIFO-Pri SchemeComparison of LIFO and FIFO
Summary
Presented a reasonably realistic model of E-commerceworkloadLIFO-Pri scheme for overload control: experimentallyverified
Server could do productive work at 60% of its capacityUpto a 7-fold increase in number of successful ‘confirm’requests when compared to single queue modelMinimal overheads
OutlookNeed to look at better indicators of overloadMore appropriate user behavior modelsAnalytical models for further insight
27 Singhmar et al Overload Control of E-commerce Web Servers
LIFO-Pri SchemeComparison of LIFO and FIFO
Thank You!
http://www.cse.iitb.ac.in/perfnet
28 Singhmar et al Overload Control of E-commerce Web Servers
LIFO-Pri SchemeComparison of LIFO and FIFO
Response Time DistributionResponse time distribution for ‘main’ page requests for ρ = 1.4
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30
P(r
espo
nse
time
> t)
Response time in sec(t)
Single Queue FIFO8Q Always FIFO8Q LIFO-Priority
29 Singhmar et al Overload Control of E-commerce Web Servers
LIFO-Pri SchemeComparison of LIFO and FIFO
Previous Work
Previous Work
Session-based admission control. (Cherkasova and Phaal)
Dynamic Weighted Fair Sharing. (Chen and Mohapatra)
Admission control with request scheduling. (Elnikety et al)
Control theory based approach. (Abdelzaher et al.)
Improving user-perceived performance at a Web server.(Dalal and Jordan)
30 Singhmar et al Overload Control of E-commerce Web Servers
LIFO-Pri SchemeComparison of LIFO and FIFO
Sample ‘Utility’ Values for Queues
Request Queue UtilityMain Page (Br-1) 27Browsing (Br-2) 22Searching (Br-3) 36Details (Br-4) 73Login (Tr-1) 3650Shipping (Tr-2) 4050Payment (Tr-3) 4500Confirm (Tr-4) 5000
31 Singhmar et al Overload Control of E-commerce Web Servers