Upload
compuware
View
267
Download
1
Embed Size (px)
DESCRIPTION
Armand Verstappen, NOC Manager at eBay Classifieds Group highlights how Compuware APM enabled eBay to protect and optimize the experience their customers love by: • Validating performance of their own datacenters • Monitoring their CDN performance • Monitoring performance trends versus their competitors To hear the full presentation, visit: http://offers2.compuware.com/APM_SAAS_ECOM_15Q1_WC_eBAY_UK_Registration.html
Citation preview
© 2014 eBay Inc. 1
Monitoring and Managing Performance
© 2014 eBay Inc. 2© 2014 eBay Inc. 2
© 2014 eBay Inc. 3© 2014 eBay Inc. 3
© 2014 eBay Inc. 4© 2014 eBay Inc. 4
© 2014 eBay Inc. 5© 2014 eBay Inc. 5
eBay Classifieds drives connected commerce by creating local marketplaces across the globe, powered
by simplicity, enriched by personal interaction.
eCG’s vision is to be the undisputed global classifieds leader
© 2014 eBay Inc. 6© 2014 eBay Inc. 6
eCG Group Maturity
Market Share
Presence Seeds High Growth Established
Canada
Denmark
Germany
UK
South Africa
SpainUS
Italy
AR
Colombia
Poland
Taiwan
Singapore
Australia
Netherlands
HK
Mexico Revenues
Develop Geo Winners
Deploy Classifieds
Full Potential
Innovate to Sustain Growth
Mobile `
© 2014 eBay Inc. 7© 2014 eBay Inc. 7
Our history of acquisitions has been a major driver of Tech complexity
eCG Growth Story
Partnerships
Source: eCG analysis
Acquired
Growth
Organic
Growth
Tech Stack
s
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
© 2014 eBay Inc. 8© 2014 eBay Inc. 8
eCG NOC
Protecting the experiences our customers love
A team working in 24x7 rotation dedicated to:
© 2014 eBay Inc. 9© 2014 eBay Inc. 9
Monitoring Through the Stack and Beyond
Outside view
Network Infrastructure
Internal Systems
© 2014 eBay Inc. 10© 2014 eBay Inc. 10
Use Case: Validating performance of own Datacenters
Alert notification arrives:
Identify ‘Own Datacenter’ as culprit by inspecting waterfall drill down (First Byte Time)
© 2014 eBay Inc. 11© 2014 eBay Inc. 11
Use Case: Validating performance of own Datacenters (continued)
Find Corresponding Public VIP, policy, pools and pool members in Observium
At the Public VIP level we establish we are not seeing a spike in requests, we’re operating well within capacity limits
© 2014 eBay Inc. 12© 2014 eBay Inc. 12
Use Case: Validating performance of own Datacenters (continued)
We only get connections and requests at pool level, we need to descend to pool member level to getTime-to-first-byte. A spike in time-to-first-byte will typically manifest itself as an increase in connections at pool level
In this example we identified a hiccup in time to first byte for one member of the pool serving this particular VIP, which trickles up in the number of open connections at the pool level. We establish there is no further site impact, and hand over to responsible operations team for root cause analysis.
We might also find an overall increase of time-to-first-byte after a new release. At release time these trends are monitored proactively, so typically we’re on top of such cases before external monitoring triggers an alarm.
© 2014 eBay Inc. 13© 2014 eBay Inc. 13
Use Case: Monitoring CDN performance of image delivery
Some of our territories do not benefit from image delivery through CDN, as our own network is ‘closer’ to the end user than any of the CDN nodes. By measuring through Gomez / APMaaS Last Mile measurements we were able to identify these cases. Based on the results, we have moved to a solution where we’re caching in our own datacenter.
As a follow up, we performed a CDN shoot out between two CDN vendors, we ended up switching vendor to improve user experience across our portfolio worldwide.
© 2014 eBay Inc. 14© 2014 eBay Inc. 14
Use Case: Monitoring CDN performance of image delivery (continued)
i.eba
yimg.
com
(2.1
6.62
.x)
i.eba
yimg.
com
(88.
221.
93.x)
i.eba
yimg.
com
(213
.241
.88.
x)
i.eba
yimg.
com
(77.
67.9
6.x)
i.eba
yimg.
com
(92.
122.
216.
x)
i.eba
yimg.
com
(2.1
6.21
8.x)
i.eba
yimg.
com
(88.
221.
92.x)
i.eba
yimg.
com
(5.1
78.4
0.x)
i.eba
yimg.
com
(90.
84.6
0.x)
i.eba
yimg.
com
(80.
157.
149.
x)
i.eba
yimg.
com
(193
.45.
10.x)
i.eba
yimg.
com
(80.
239.
228.
x)
i.eba
yimg.
com
(217
.212
.225
.x)
i.eba
yimg.
com
(80.
239.
205.
x)
i.eba
yimg.
com
(2.1
6.54
.x)
i.eba
yimg.
com
(184
.84.
182.
x)0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
14.00%
16.00%
18.00%
ePS performance in Germany measured from Backbone Nodes
Average Response Time (sec) Percentage of hits
© 2014 eBay Inc. 15© 2014 eBay Inc. 15
Use Case: Monitoring CDN performance of image delivery (continued)
i.eba
yimg.
com
(2.1
6.21
8.x)
i.eba
yimg.
com
(88.
221.
93.x)
i.eba
yimg.
com
(88.
221.
92.x)
i.eba
yimg.
com
(77.
67.9
6.x)
i.eba
yimg.
com
(5.1
78.4
0.x)
i.eba
yimg.
com
(217
.212
.225
.x)
i.eba
yimg.
com
(79.
140.
81.x)
i.eba
yimg.
com
(149
.3.1
80.x)
i.eba
yimg.
com
(92.
122.
216.
x)
i.eba
yimg.
com
(92.
122.
124.
x)
i.eba
yimg.
com
(184
.25.
107.
x)
i.eba
yimg.
com
(88.
221.
212.
x)
i.eba
yimg.
com
(46.
33.7
5.x)
i.eba
yimg.
com
(90.
84.5
4.x)
i.eba
yimg.
com
(165
.254
.134
.x)
i.eba
yimg.
com
(96.
17.1
5.x)
i.eba
yimg.
com
(213
.155
.157
.x)
i.eba
yimg.
com
(46.
33.7
6.x)
i.eba
yimg.
com
(46.
33.7
2.x)
i.eba
yimg.
com
(204
.2.2
22.x)
0
2
4
6
8
10
12
14
16
18
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
ePS performance in Germany measured from Last Mile
Average Response Time (sec) Percentage of hits
© 2014 eBay Inc. 16© 2014 eBay Inc. 16
Use Case: Monitoring Performance trends vs. Competitors
Weighed Page Load times for Home Page, Search Results Page and Item detail pages are reported to the business on a monthly basis.
Downside: Current method is labor intensive; Competitor tests need frequent repair as items expire.
© 2014 eBay Inc. 17© 2014 eBay Inc. 17
Future steps
- Move to Last Mile tests for competitor comparisons- Speed up the correlation process:
- Push Observium data into Graphite (feature development sponsored by eBay)
- Use Gomez / APMaaS API to pull performance data into Graphite
- Have custom text in Gomez / APMaaS Notifications to have direct url pointing to relevant Graphite graphs
- Create custom reports to help keep third party advertisers honest
- Create customer reports to aid in monthly site speed reporting
- Measure Mobile API call latency from Gomez / APMaaS
© 2014 eBay Inc. 18© 2014 eBay Inc. 18
Questions?