March 2014

Case Study: Using WSO2 API Manager to deliver dynamic

display advertisingChuck MegivernSenior Software EngineerMyWebGrocer, Inc.cmegivern@mywebgrocer.com

May 2014

March 2014

Chuck Megivern is a Senior Software Engineer for MyWebGrocer, Inc.

He is the company technical lead for open source technologies. Chuck has been developing advertising applications for MyWebGrocer and before that, Dealer.com, for the last five years.

He has delivered Display Advertising, SEM, SEO and Digital Video applications and APIs. He has also delivered multiple cloud based server deployments.

Earlier in his career, Chuck developed software for IBM and also had a number of engineering assignments there.

Chuck has a BSEE degree from Lehigh University, and an MSEE degree from Syracuse University.

About the author

March 2014

• MyWebGrocer (MWG) was founded in 1999.• We serve the grocery industry and consumer packaged goods industry. • We provide an online grocery platform used by leading retailers

encompassing over 10,000 retail locations in the US. • Expanding in Asia Pacific and Europe.

• We also have extensive eCommerce and eMarketing solutions. • We provide digital display advertising on our own ad network, as well as on

public internet sites that participate in other online ad networks, such as Google DoubleClick.

• My area provides APIs for internal company use, and for external business partners.

Introducing MyWebGrocer

March 2014

• Our case study involves using WSO2 API Manager to enrich display advertising served to consumers via the Google DoubleClick ad network.

• The ad network delivers an Adobe Flash based Rich Media ad to consumer’s desktops while they are browsing internet sites.

• The ad campaign lasts for a number of weeks. The Rich Media ad, using Action Script, calls a MWG API and dynamically pulls in data every time the ad is displayed.

• The API delivers product information that updates daily. That data • comes from the retailer’s store that is closest to the consumer, and

• is currently advertising the products of interest.

• Most consumer advertising application use cases involve responding to a consumers 'click' expressing interest.

• However, this high traffic use case invokes the API every time that a display ad is presented to a consumer.

• This presentation discusses our technical approach and application stack used to handle this high volume.

Our case study

March 2014

WSO2 Product Usage

• Case study usage • API Manager 1.5• Business Activity Monitor (BAM) 2.4.0

• Current Usage• API Manager 1.6• Business Activity Monitor (BAM) 2.4.1

Types of APIs served

• Case study: very high volume consumer browser API invokations.• At the same time, this distributed API manager installation serves:

• external business partners using MWG API’s• internal MWG servers invoking MWG API’s• Heterogeneous API consumers and API implementing servers

• Java• Java related technologies• Microsoft .NET

• WSO2 is helping us be a “connected enterprise”

MWG WSO2 Product Usage and the MWG API environment

March 2014

WSO2 API Manager - Composition

March 2014

Three APIM Gateways. Two Primaries, one is a backup without a BAM connection. The F5 has an API health check

nginx: preprocesses static contentLoad balances application servers (APIM 1.5)

API sync: rsync between gateways(not shown)

MySQL DBs withCircular replication (not shown)

F5 load balancer exposed to public internet

APIM Store / Publisher not shown,Not accessible from public internet

Our WSO2 Product Deployment

March 2014

• Consumer visits a website that is showing ads.• Doubleclick network delivers the ad to the

consumer’s browser. There are multiple ads that can be displayed. Each ad contains a set of keywords relating to the item advertised, and the retailer id.

• Rich media ad determines the consumer’s zip code.• MWG API is called with the zip code, keywords,

retailer id.• The API returned the nearest store and information

on a relevant item for sale. The consumer sees …

An ad is shown: step by step process

March 2014

March 2014

• Geolocation query for a particular retailer.• bounding box by latitude, longitude• Haversine function to sharpen results• Returns a list of stores, ordered by increasing distance.

• Store circulars are queried • keyword search

• Caching: database retrieval results• geolocation query caching• store circular data caching• memcached used

Database loaded via background E-T-L processes

API functionality

March 2014

• API Manager requires an Authorization header. • The Ad Unit, running Adobe ActionScript, will supply this header only with a

POST command.

• ActionScript also supports client security by demanding a crossdomain.xml file.GET host:port/crossdomain.xml (no context root)

• nginx is used to intercept all the http calls • serves the static crossdomain.xml file• passes all other requests to API Manager. • also used: load balance application server endpoints (needed for APIM 1.5)

Initial Technical Challenges

March 2014

• Received only a total impression count from marketing. No peak load estimate.

•API Manager / BAM integration vital.

• Essential: Used summarized BAM statistics from the outset.

•Discovered with BAM: Workload was not random, or average. but very “bursty”.

• Bottleneck was our serving infrastructure, not API manager.

• Added a memcached infrastructure on the fly.•essential to boost for application server response time.

Performance challenges

March 2014

• Uncached API service time is 150 – 200ms.• Object caching brings to 22ms average service time.• Cache hit ratio: 85 – 90%.• Record (with fast average service time): 376

responses per second at 5ms average service time.• Peak responses per second: 402 responses /second

(our throttle).

Performance highlights

March 2014

• A number of traffic bursts: 100 to 400 responses / second.

• Traffic slows in the overnight.• Traffic peaks in late afternoon, evening.• Hourly, traffic slows 25 minutes after the hour. It

can increase 50% at 55 minutes after the hour.• Assume: ad network serving hourly adjustments or TV commercials.

• Overall mean: 4.5 responses per second.• 90th percentile: 21 responses per second.• Note: performance summarized by minute, based

on the BAM analytics toolbox for API manager.

Performance Highlights, cont’d.

March 2014

• This plot range: up to 100/sec

Responses per second distribution

March 2014

• This plot range:100-400 per sec.

• High end bursts• Assume:

Ad network• Few instances.

Responses per second

March 2014

Can spike if:• Not enough processing

threads• Cache entries expired.• Many cache misses and

high traffic.• No API Manager

limitation seen.• Note: uncached service is

150ms – 200ms.• With caching design

point, database is not limiting.

Average Service Time

March 2014

• WSO2 BAM Cassandra storage• Cassandra used as a ‘fast write’ repository for the streams of data

from the API manager gateways.• BAM Hive scripts summarize on a per-minute basis, put summaries in

a relational db for reporting.• Cassandra data does not seem to expire for us, so the Hive scripts

keep resummarizing.• Purging old, already summarized Cassandra data.

• WSO2 Api Manager: Publisher• Provides cool displays of usage right in the publisher from the

summarized statistics.• Publisher had trouble handling the data once we had a lot of data.

We turned off the display inside Publisher.

Issues Encountered

March 2014

• Key Features• API Design Wizard

• Support for OAuth2 Scopes

• Statistics Dashboard for developers

• Multi tenant workflow support

• Workflow integration for token generation

• URL templating support for API definition

• Customisable Store themes for tenants

• Forum Support - API Store

• Visibility Control for Documentation

• Statistics dashboard for southbound API calls

• Using Windows Authentication for OAuth2 Token generation

API Manager 1.7.0 is coming up...

March 2014

• The WSO2 API Manager platform is well suited to handle high volume traffic. It was very capable and adaptable.

• Essential: WSO2 Business Activity Monitor, using the WSO2 provided API Manager BAM Analytics toolbox,allowed us to

•Understand our traffic patterns•Tune our server infrastructure•Produce metrics for reporting

Conclusion

March 2014

• Thanks for listening.• Thanks: Ad unit graphic development: Kurrent

Interactive: http://www.kurrent.com/• Thanks: WSO2 for a flexible configurable product

set, based on open source standards.

Thanks!