Data Warehousing Infrastructure on Cloud

Akamai Confidential

Data warehousing infrastructure on Cloud

Praveen HebbagodiDirector of Engineering, Akamai Technologies

Akamai Confidential ©2011 AkamaiPowering a Better Internet

Agenda

• Introduction to Akamai• Akamai BI Solutions• Data warehousing platform

• Features• Architecture• Operations

• Conclusions


The Akamai Network…

- 88,000+ Servers- 1,100+ Networks- 1,600+ Locations- 650+ Cities- 71 Countries

Accelerating Daily Traffic of:• 3+ Tbps• 11+ million hits per second• 10+ million concurrent streams• 800+ billion deliveries/day• 30+ petabytes/day

Connecting:• 465 million unique IP addresses• From 234 countries

Deflecting Attack Traffic:• From 198 countries• Targeting 10,000 unique ports

… a large-scale on-demand distributed computing platform


Major services provided by Akamai

Content delivery:• HTTP/S (15-30% of total HTTP traffic!)• Live and On-Demand Streaming

Application delivery:• Web Application Acceleration• Dynamic Site Acceleration• EdgeComputing• IP Application Acceleration

Example applications:• Online commerce, media delivery,

B2B/B2C applications, software downloads,social networking sites, …

• You likely use many of our services each day


Akamai BI Solutions

Example Solutions:

Media Analytics

• A comprehensive solution for content and audience intelligence for broadband media

• Features

• Dashboards for Engagement Overview, Ad Optimizations and Content Usage

• Standard Reports with detailed engagement and audineceinformation

• Custom dimensions and reports to suit business specific needs

Helps our customers to get deeper insights into their audience and content usage in the context of their business


Akamai BI Solutions: Examples

QOS Monitor

• Real-time quality of service monitoring solution for online media delivery

• Features

• Set thresholds for breaches, find root causes and resolve issues using “Notifications”

• Live Real-time monitoring console with data aggregation as fine as 30 sec.

• 20 standard reports & dashboards for historical diagnostics & debugging.


Features

• Intuitive dashboard & report builder UI with advanced visualizations

• Over 50 Standard dimensions & metrics• Support for any customer specific dimensions

• Regular expression extraction• Plug-in API

• Ad-hoc query, drill-down• Lookup tables• Dashboards & reports can be provisioned on the fly• Real-time notifications• Data access via web interface, SOAP API, Email & download

(CSV, PDF, HTML)


Analytics workflow programmable via portal• Data sources, filters, metrics, dimensions, reports, dashboards configured via

xml metadata

Distributed data collection in the Cloud• Data sources: end user machines (beacons), edge server logs, agents• Filtering, and partial aggregation at the source and in collection layers• Facilitates scalability and better utilization of resources

Data warehousing Platform

Reporting Data Processing

Data Storage

xmlxml

Data Sources

Data Collection

xml


Data Processing is a flexible map-reduce framework• Dataflow graph of map-reduce operations• Enhancements for better latencies, scheduling optimizations • Faster message passing interfaces (network, in-memory)

Data warehousing Platform

Reporting Data Processing

Data Storage

xmlxml

Data Sources

Data Collection

xml


Data abstraction is a set of data cubes• Supports fast slice-and-dice, drill-down operations,…

Data cubes are physically realized in distributed columnar DB• SQL interface, column compression, bitmap indexes• In-situ updates, write-optimized store• Sharding and cluster management

ReportingData Storage

Time

Artis

t Nam

e

xmlProcessing

Data-warehousing Platform


Approach to Operations

• Treat failures as normal

• Build in layers of redundancy• At all levels: geo/network, with in a cluster• Multi-path communications

• Weaker data consistency models

• Zoning • Dynamic Configuration• Software Installs

• Design systems that run themselves• Autonomic response where appropriate


Conclusions

• Being on cloud facilitates in building highly scalable platform for “big data” applications

• Design for failures• Build redundant systems at all levels• Multiple levels of fault-tolerance

• Automation, autonomics, more automation…• Avoid “manual changes”

• They will happen, so have good process to minimize/track

• Deterministic software and config management system• Converges to consistent state & built-in safe roll-back

• Good tools for understanding system behavior and data quality

• Sophisticated tools for capacity management and performance monitoring


To find out more about Akamai…

More info: www.akamai.com

Contacting me:

Praveen [email protected]

Technical publications:http://www.akamai.com/html/perspectives/techpubs.html

Jobs: http://www.akamai.com/html/careers/index.html

http://twitter.com/akamaijobsindia

Questions?

Akamai Confidential

Thank you

Technology

Data Warehousing Infrastructure on Cloud