Understanding Business APIs through statistics

Understanding Business APIs through Statistics

Amila De SilvaDinusha Senanayaka

About WSO2

• Providing the only complete open source componentized cloud platform

– Dedicated to removing all the stumbling blocks to enterprise agility– Enabling you to focus on business logic and business value

• Recognized by leading analyst firms as visionaries and leaders– Gartner cites WSO2 as visionaries in all 3 categories of

application infrastructure– Forrester places WSO2 in top 2 for API Management

• Global corporation with offices in USA, UK & Sri Lanka– 200+ employees and growing

• Business model of selling comprehensive support & maintenance for our products

150+ globally positioned support customers

Overview

● Role of Statistics in API Management○ Statistics to understand user communities○ As tool for making strategic decisions.

● Different Options for analysing data○ Phases in Data analysis○ Offline vs Real-time analysis

● BAM as an offline data analytic engine○ How large event streams gets summarized

● CEP for real time processing○ Architecture of CEP○ Stages of an execution plan

● GA as a feedback channel for app developers● Options for visualising data.

○ Generating reports out of BAM summarized data.

APIs - the way to go

● APIs are no longer a luxury - they are becoming a necessity.

● API - A common interface for disparate platforms

● More open you are more users you will attract.

● Build an ecosystem around your APIs.

Knowing your users

Why do you need statistics?

● There are many connected components● One wrong move can make disastrous

consequences.● Feedback channels are important -

○ See how users access them and learn - It explains what they say.

● Discover hidden potentials within your communities○ Going for a new billing model - See how users

access them.● Can you check them all manually?

What we provide

● Two branches in analysis○ Offline Data analysis○ Realtime Analaysis

● Offline Data Analysis○ Gives you a general idea of your APIs○ Who your users are.○ Leads you to make long term decisions.

● Realtime analysis○ Identifies co-occurrence of several useful business

flows.○ Find and act on it.

How we analyse it

● Done in two main phases○ Data Gathering○ Analysis.

● Gather data at product level.○ multiple event streams for request, response and faults○ each invocation as an event

● Delegating analysis to a different product○ Simply any tool capable of processing event streams can

analyse data.○ Currently BAM and CEP are used.

● Extensions to achieve product specific analysis○ In BAM this is done through a hive script○ In CEP this can be done by a transport adaptor + execution

plan.

Data Gathering

● Event = Invocation● In the gateway we collect data related to

each invocation● Three different event streams

○ Request Stream○ Response Stream○ Fault stream

Structure of a StreamRequest Stream

'payloadData':[ {'name':'consumerKey','type':'STRING'}, {'name':'context','type':'STRING'}, {'name':'api_version','type':'STRING'}, {'name':'api','type':'STRING'}, {'name':'resource','type':'STRING'}, {'name':'method','type':'STRING'}, {'name':'version','type':'STRING'}, {'name':'request','type':'INT'}, {'name':'requestTime','type':'LONG'}, {'name':'userId','type':'STRING'}, {'name':'tenantDomain','type':'STRING'}, {'name':'hostName','type':'STRING'}, {'name':'apiPublisher','type':'STRING'},

{'name':'applicationName','type':'STRING'}, {'name':'applicationId','type':'STRING'} ]

Response Stream

'payloadData':[ {'name':'consumerKey','type':'STRING'}, {'name':'context','type':'STRING'}, {'name':'api_version','type':'STRING'}, {'name':'api','type':'STRING'}, {'name':'resource','type':'STRING'}, {'name':'method','type':'STRING'}, {'name':'version','type':'STRING'}, {'name':'response','type':'INT'}, {'name':'responseTime','type':'LONG'}, {'name':'serviceTime','type':'LONG'}, {'name':'userId','type':'STRING'}, {'name':'tenantDomain','type':'STRING'}, {'name':'hostName','type':'STRING'}, {'name':'apiPublisher','type':'STRING'}, {'name':'applicationName','type':'STRING'}, {'name':'applicationId','type':'STRING'} ]

Fault Stream

'payloadData':[ … {'name':errorCode','type': 'STRING '}, {'name':‘errorMessage','type': 'STRING '}, .. ]

Analysing Data with BAM...

Analysing Data with BAM

● Used for offline Analysis.● Aggregates, stores, then analyses data.● Capable of receiving events from multiple data

agents.○ In APIM 1.5.0 we use a load balancing data publisher.

● All the incoming events are first written to a Cassandra DB

● Data analysed using a hive script running on top of a hadoop cluster.

● Summarised data written back to a Relational DB.


Summarized tables


Table Descriptions


Visualizing summarized data

Analysing with BAM...

Summarized data usages for billing

● Decide on a billing model● Integrate with existing billing engines

Real-time Analysis with CEP

● APIM can publish events to multiple data agents.

● Same streams are used to publish events to CEP

● Whenever it receives a new event CEP executes a query and evaluates the event.

● Siddhi queries define the evaluation criteria.

CEP Architecture

CEP Architecture...

● Transport Adaptors to read/write event sources.○ We can create alternate event streams out of the

original streams.● Event builders convert events to a standard

format.● Queries defined as query plans.● Event processor executes a query plan.

○ It might generate additional events.● Event formatters convert back the output.

Example

● Task : Identify client applications sending too many repeated requests○ Too Many : 5 requests ○ repeated : within 1 minute

● How to identify a client○ Consumer key

And the query is:from AMRequest#window.time(1 min)insert into outStreamcount(consumerKey) as myCount,api_version,api,consumerKeygroup by consumerKey,api_version,apihaving (myCount > 5);

Flow in CEP...

● Transport adaptor reads requestStreamDefn● Creates a second input stream AMRequest

○ This only contains subset of data.● Event builder transform the input stream● Query is defined in a query plan.

○ Even processor executes each of the registered query plans.

● Upon meeting the condition an event is generated.● Event formatter converts the event.● Mail is sent through the transport adaptor.

Monitoring Statistics with Google Analytics

• API Manager can be easily configured with Google Analytics

<GoogleAnalyticsTracking><Enabled>false</Enabled>

<TrackingID>UA-XXXXXXXX-X</TrackingID></GoogleAnalyticsTracking>

Monitoring Statistics with Google Analytics

Visualising

● Needed for interpreting statistics.● Gadgets provided in BAM● Plugin any external tool for generating reports● Combining summary tables to create new

views.

Summary

• Statistics as a channel to verify business objectives.

• WSO2 API Manager provides different mechanisms to analyze data.

• WSO2 BAM can be integrated with API Manager process offline data.• Gather -> Cassandra• Analyze -> Relational DB

• WSO2 CEP can be integrated with API Manager and real time data analyzing.

• Google Analytics integration with API Manager.

Engage with WSO2

• Helping you get the most out of your deployments• From project evaluation and inception to development

and going into production, WSO2 is your partner in ensuring 100% project success

Technology

Understanding Business APIs through statistics