Big Data & Analytics Architecture

Advanced Analytics Platform Deep DiveComponents, Patterns, Architecture DecisionsISA-3637 (Tue Nov 5 11:15 AM – 12:15 AM)

Dr. Arvind Sathi [email protected]

Richard Harken [email protected]

Tommy Eunice [email protected]

Mathews Thomas [email protected]

© 2013 IBM Corporation

Please note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

Acknowledgements and Disclaimers

Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates.

The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

© Copyright IBM Corporation 2013. All rights reserved.

•U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

•Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2, Maximo, Clearcase, Lotus, etc

IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBMtrademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml

If you have mentioned trademarks that are not from IBM, please update and add the following lines:

[Insert any special 3rd party trademark names/attributions here]

Other company, product, or service names may be trademarks or service marks of others.

http://www.ibm.com/legal/copytrade.shtml

Content

• Use cases to support Business Architecture

• Components to support Application Architecture

• Data Integration

• Privacy Management & Archiving

• Location & Lifestyle Analytics

• Adaptive Analytics

• Momentum and Conclusions

Industry

ImperativesCreate & Deliver

Smarter Services

Transform

Operations

Build Smarter

Networks

Personalize

Customer

Engagements

MAJOR use

cases

Emerging

Use Cases

Smarter Advertising

Customized Customer

Marketing

3rd Party API’s

Cloud services for

SMEs, enterprises

Contactless services

(payments and banking)

M2M (smart cars, e-

Health)

Tiered Services

Big Data Scale

Investment Decisions

Lower storage

requirements

Smarter Returns

Analyze data before it

lands – then store only

what you need

New analytic models

Share critical

information across the

enterprise vs. deliver

multiple copies of the

data

Traditional

Infrastructure

Optimization

Product Knowledge Hub

Content Network

Distribution

Proactive Device

Management

Network Fault Prevention

ICTO (Energy Savings)

Real Time Traffic

Optimization

Network Abuse from

excessive data users

Discrete on-line charging

for quality of experience

Real time automated

capacity management for

dropped calls

SON Capacity

Management for special

events (traffic offload)

Service Migration

Social Advocacy

Cross Offering

Transparency

Smarter Customer

Interaction &

Engagement

Real-time Customer

Experience Insight

Smarter Campaigns

Customer Retention

Micro Segmentation

Marketing

Next Best Offer

Retail cross Channel

optimization

Location Based Services Pro Active Call Center

Customer Data/Location Monetization

Smarter Campaigns

Customer Knowledge Hub

Social Media Insight

IT Infrastructure Transformation (Traditional to Big Data)

Product Knowledge Hub

Voice & Data Fraud

Network Analytics

Network Infrastructure Planning (Performance, Capacity, Usage)

AAP – Telecommunications Use Cases

Cross Industry Solutions

How to turn streaming noisy Telco Location data into meaningful location, then discover customer insights

Call Detail Records

SMS Voice

GPS Tracking

Cell Tower

Wifi AP Maps

GIS, POI

Special Service

Numbers

e.g bank, 1-800

Reference Data

Stream data

subscriberId:Timestamp: Position: latitude + longitudePrecision: 0~2 kmDirection: nullableSpeed: nullableActivity : nullable

Analyzable Location Event DataWho, when, where and what

Meaningful Location

subscriberId:home: Work: POIs & period …Sequence of meaningful Locations… Commute means: car/subway/bus

Micro segmentaton

Business travelerRegular commuter Heavy driverSocial ButterflyMom…..

Every Sunday noon, Bob goes to xxx mall to shopping and has lunchEvery Thursday afternoon, Bob goes to customer site at XXX…..

Location Patterns on Individual and Group level

Mobile Location Data Processing: Map mapping,

Business rules et.

Big Data Integration

Spatio-Temporal Event Association Analysis

Wifi off load

Location Pattern Analytics

Mobile Couponing Use Case

Customer Action Cuppa Heaven/Offertel Action

2) Opts-in to receive mobile coupons from the Telco

Advanced Analytics Platform

TelcoCustomer Profile

1) Contacts Offertel Communications to run campaign for a new store next to a movie theater

7) Monitor Campaign Performance

4) Driving habits, coffee preference, & opinion leaders used to prioritize customer target list

Campaign Delivery System

5) Priority list transferred to conduct campaign

Telco clients who have opted out of Mobile

coupons

6A) Receives mobile coupon for new Cuppa Heaven store

6A) Receives mobile coupon for new CuppaHeaven store

6B) Deliver Coupons to mobile opt-out clients via email & web site

3) Use Social media to establish“Opinion Leaders”, potential coffee drinkers, movie goers

7) Posts on twitter, Facebook public fan page for CuppaHeaven

Organizational FOCUS areas

MAJOR use cases

(sales play)

Industry Teamuse cases

Create differentiated customer experiences Build an agile digital supply chain

“Connected Consumer” “Smarter Media”

Advertising Optimization

Operations Analysis & Optimization

Business Process Transformation

Infrastructure Mgmt & Security

Audience & Marketing Optimization

360o View of the Customer

Multi-Channel Enablement

Customer & Market Insight

•Social Profiling/ Sentiment Analysis •Churn Optimization•Customer Care Optimization•Audience/ Viewing Duplication•Audience Composition Index•Multi-Platform Ad Performance•Advertiser Revenue Analysis•Real Time Audience Targeting•CRM Optimization

•Real-time ad targeting •Ad inventory Optimization•Real-time ad reporting •Search engine optimization•Campaign optimization (in-flight) •Marketing campaign effectiveness•Network & infrastructure optimization•Network Demand Forecasting•Content optimization •Content demand forecasting•IP Rights Optimization

Media, Metadata & Optimization.Digital Commerce Optimization

AAP – Media and Entertainment Use Cases

AAP for Real-time Bidding of Advertisements

TURN DMP

TURN DSP

Telco Website

Flex Tag

Campaign Details

Campaign Mgmt

TurnTelco

Bid Req

Offer &Response

Bid Req

Offer & Response

CampaignFeedback

TelcoData

Additional data (e.g. Offer acceptance, location)

Customer Data

ContentProvider

Real-timeScoring

Predictive Models

Data Integration

AnalyticsVisualization


Customer

Location

Events / xDR

Usage

Content








New Architecture to Leverage All Data and Analytics

Data inMotion

Data atRest

Data inMany Forms

Information Ingestion and Operational Information

Decision Management

BI and Predictive Analytics

Navigation and Discovery

IntelligenceAnalysis

Landing Area,Analytics Zoneand Archive

Real-timeAnalytics Video/Audio Network/Sensor Entity Analytics Predictive Exploration

,Integrated Warehouse, and Mart Zones

Stream Processing

Data Integration Master Data

Streams

Information Governance, Security and Business Continuity

IBM Big Data Advanced Analytics Platform (AAP) Architecture

A

B

C

D

G

AAP Capabilities

High Performance Historical analysis

Model Based Predictive Analytics

Real-time scoring, classification, detection and action

Visualize, explore, investigate, search and report

High Performance Unstructured Data analysis

Discovery Analytics

Take action on analytics

F

Information Interaction

Analytics Engine

Prediction / Policy Engine

Sense, Identify,

Align

Reports

Geo/Semantic

Mapping

Dashboards

Simulation

Outcome Optimization

Model Creation

Semi Structured

Data

Data

Repositories

Network Events

Network Policies

Continuous F

eed

Sourc

es

XDR

Batch Data

Data for Historical Analysis

Deploy Model

Streaming Engine

Streaming Data Categorize, Count, Focus

Score, Decide

Historical Data Models

In Database Mining

Reports & Dashboards

Ad-hoc Queries

Actions

Event Execution

Policy Mgmt

Exte

rnal

Data

Social

3rd party

High Velocity

High Volume

Open API

Customer Activities

A

B

C

DG

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Campaign Mgmt.

Pro-active Customer

Experience Management

Pro-active Network Mgmt

Real time Scoring & Decision Mgmt.

...

Deploy Model

Policy Management

Data Integration ETL

Deduplicate

Standardize

Identity Resolution

Network Topology

Data

Application & Usage

Data

Customer Data

Capture Changes

Un-Structured

Data

HadoopE

E

Structured Data

InsightFSearch, Pattern Matching, Quantitative, Qualitative

Enterprise Data Warehouse


Create & Deliver Smarter Services Transform Operations

Build Smarter Networks

Personalize Customer Engagements

Database Server


A

B

C

D

G

AAP Capabilities






Discovery Analytics


F


Analytics Engine


Sense, Identify,

Align

Reports

Geo/Semantic

Mapping

Dashboards

Simulation


Model Creation

Semi Structured

Data

Data

Repositories

Network Events

Network Policies

Continuous F

eed

Sourc

es

XDR

Batch Data


Deploy Model

Streaming Engine


Score, Decide


In Database Mining


Ad-hoc Queries

Actions

Event Execution

Policy Mgmt

Exte

rnal

Data

Social

3rd party

High Velocity

High Volume

Open API

Customer Activities

A

B

C

DG

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Campaign Mgmt.

Pro-active Customer




...

Deploy Model

Policy Management


Deduplicate

Standardize

Identity Resolution

Network Topology

Data

Application & Usage

Data

Customer Data

Capture Changes

Un-Structured

Data

HadoopE

E

Structured Data







InfoSphere Streams

SPSS

WODM, Optim

PDA

Social Media Analytics

InfoSphere Data Explorer

Cognos

InfoSphere BigInsights

IBM (Unica)

Campaign

WODM

PDOA

SPSSDatabase Server

BPM

Data StageQuality Stage

MDM

Capabilities OverviewCapability Capability Description

Align diverse streams of data, identify customers, align to IDs, sense data importance

Categorize incoming data, use window counts to aggregate atomic data or threshold vioilations,

focus attention on monitored situations abstracted from raw events

Use scoring models developed by prediction engine to score observations, activities, customers,

etc. in real time

Make data ready for execution of events – e.g., designing campaign messages based on

information available.

Includes TEDA and geo-spatial accelerators

Create models using historical data sources

Optimize outcomes by promoting best model for a particular treatment (Champion / Challenger)

Manage policies associated with decisions – e.g., WODM decision rules, Optim data policies, etc.

Includes SPSS Deployment Server

Includes SPSS location analytics

Provide capabilities for storage of structured, unstructured and semi-structured data

Provide capabilities for analytics using DB functions (e.g., SPSS model development)

Provide capabilities for data archival using archival policies

Includes Optim / DS for archival policy execution

Deep analysis of consumer behavior is performed to mine data for model creation

Includes unstructured search, pattern matching using arbitrarily defined patterns, qualitative

analytics, quantification of data (e.g., sentiment analysis)

Includes Big Insights accelerators

Perform Ad hoc queries, standard reports, dash board

Run simulation models, what-if analysis

Geo-spatial and semantic viewing of data

Streaming Engine


Database Server

Insight


AAP Capabilities

Content








16

Mature Organizations are Looking for Instantaneous

Insight from Data

Speed to insight

Total respondents n = 973

Respondents were asked how quickly business users

require data to be available for analysis or within

processes. Box placement reflects the prevalence of

that requirements within each a stage.

17

Current fact finding

Analyze data in motion – before it is stored

Low latency paradigm, push model

Data driven – bring data to the analytics

Historical fact finding

Find and analyze information stored on disk

Batch paradigm, pull model

Query-driven: submits queries to static data

Traditional Computing Stream Computing

Stream Computing Represents a Paradigm Shift

Real-time Analytics

18

Massively scalable stream analytics

Linear Scalability

• Clustered deployments –

unlimited scalability

Automated Deployment

• Automatically optimize

operator deployment

across nodes

Performance Optimization

• Parallel & pipeline

operations

• Efficient multi-threading

Analytics on Streaming Data

• Analytic accelerators for a

variety of data types

• Optimized for real-time

performance

Visualization

Streams Runtime

Deployments

Sink

Adapters

Analytic

Operators

Source

Adapters

Automated and

Optimized

DeploymentStreaming Data

Sources

Streams Studio IDE

19

ModifyFilter / Sample

Classify

Fuse

Annotate

Big Data in Real Time with InfoSphere Streams

Score

Windowed Aggregates

Analyze


A

B

C

D

G

AAP Capabilities






Discovery Analytics


F


Analytics Engine


Sense, Identify,

Align

Reports

Geo/Semantic

Mapping

Dashboards

Simulation


Model Creation

Semi Structured

Data

Data

Repositories

Network Events

Network Policies

Continuous F

eed

Sourc

es

XDR

Batch Data


Deploy Model

Streaming Engine


Score, Decide


In Database Mining


Ad-hoc Queries

Actions

Event Execution

Policy Mgmt

Exte

rnal

Data

Social

3rd party

High Velocity

High Volume

Open API

Customer Activities

A

B

C

DG

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Campaign Mgmt.

Pro-active Customer




...

Deploy Model

Policy Management


Deduplicate

Standardize

Identity Resolution

Network Topology

Data

Application & Usage

Data

Customer Data

Capture Changes

Un-Structured

Data

HadoopE

E

Structured Data







InfoSphere Streams

SPSS

WODM, Optim

PDA



Cognos


IBM (Unica)

Campaign

WODM

PDOA

SPSSDatabase Server

BPM

Content








What is Sensitive Data

Personally Sensitive

• Information that can be misused to harm a person in financial, employment or social way. (Names, Social Security Number, Credit Card, etc.)

Network Sensitive

• Information that can be misused to breech or disable critical network communication (Circuit Identifiers, IP Addresses, etc.)

Corporate Sensitive

• Information that can misused to compromise the competitive position of a company (Operational Metrics, etc.)

6 steps that work together to achieve an acceptable and manageable level of data security

Processes &

Information assets

Audit

Manage

Define process

Implement

Controls

Assess Risk

Data masking requires a combination of process, templates and tools

Our approach brings together data masking infrastructure using DataStage and

ProfileStage, combining with Masking on Demand plug-in using Optim

technology.

InfoSphere Analyzer Optim, DataStage

Tools

Templates

Masking Utilities- Incremental Autogen

- Swap

- Relational Group Swap

- String Replacement

- Universal Random

Data Definitions- Customer ID

- Name

- Address

- Credit Card No

- Social Sec No

- Etc.

Identify Select Verify Implement

Reusable Processes

Validate


A

B

C

D

G

AAP Capabilities






Discovery Analytics


F


Analytics Engine


Sense, Identify,

Align

Reports

Geo/Semantic

Mapping

Dashboards

Simulation


Model Creation

Semi Structured

Data

Data

Repositories

Network Events

Network Policies

Continuous F

eed

Sourc

es

XDR

Batch Data


Deploy Model

Streaming Engine


Score, Decide


In Database Mining


Ad-hoc Queries

Actions

Event Execution

Policy Mgmt

Exte

rnal

Data

Social

3rd party

High Velocity

High Volume

Open API

Customer Activities

A

B

C

DG

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Campaign Mgmt.

Pro-active Customer




...

Deploy Model

Policy Management


Deduplicate

Standardize

Identity Resolution

Network Topology

Data

Application & Usage

Data

Customer Data

Capture Changes

Un-Structured

Data

HadoopE

E

Structured Data







InfoSphere Streams

SPSS

WODM, Optim

PDA



Cognos


IBM (Unica)

Campaign

WODM

PDOA

SPSSDatabase Server

BPM

Content








© 2012 IBM Corporation

Buddies, Hangouts, Globtrotters

Areas of mobility analytics

Individual Lifestyle and Usage profiles

Popular Locations with specific profiles

Who are the Buddies

Predicting where people go

Who Are You?HomebodyDaily GrinderDelivering the GoodsGlobetrotterSofa Surfer

10 Top Hangouts

Mobile ID Buddy Rank

2702 1

1256 2

8786 3

4792 4

8950 5

What are Profiles

• Lifestyle Profiles are defined by marketing analysts for specific use cases or marketing programs

• Usage Profiles are created using data mining algorithms and define how a person uses services during the day

• Location Affinity is created with algorithms and determines preferred locations for individuals throughout the day and week

• Together these uniquely define a person with relation to how the retailer or marketer might want to market to them

Creating Groups of Mobility Profiles Enables Better Prediction for Certain Groups

profiles breakdown like this

Homebody, doesn't visit too many unique locations

Daily Grinder, back and forth to work, quiet weekends, makes stops along the way

Norm Peterson, inside the lines, no deviations

Delivering the goods, no predictable patterns, many different locales during the day

Globe Trotter, either not in town, or keeps their phone turned off

Rover Wanderer, spends evenings at various location (sofa surfers www.couchsurfing.org)

“Other”, is a group hard to categorize

By Profile, when is it easy or difficult to predict where they will be?

Profile Day Time Predictability

Daily Grinder Thursday Dinner Highest

Daily Grinder Friday Afternoon Lowest

Homebody Saturday Night Highest

Homebody Wednesday Morning Lowest

These are the 2 most predictable profiles, yet there is diversity in their predictability.To best communicate with Daily Grinders, contact them on Thursday Afternoons just before dinner

Preferred Locations of by profile type at Lunchtime Weekdays (Central Stockholm)

Delivering the Goods

Night Shifters

Daily Grinders

What analysis is available (Anonymous Data)

From the mobility profiles, summarized, anonymous analysis is available

Summarized to ensure anonymity, analysis of popular locations by time of day and profile of subscribers is possible

For retailers this information can help understand what types of people are nearby at lunch time

What types of people prefer which areas. Some obvious results are Globe Trotters go to airports, Daily Grinders go to office buildings. Other non-obvious results show up also.

Are there predictable patterns that we can use to target certain groups in the future?

What Makes this Possible?

Using the power of Netezza and modeling capabilities of SPSS we can literally throw all the data at data mining algorithms and create discrete clusters of subscribers by activity, mobility

Apply the data mining outputs to the entire subscriber base by creating detailed specific analyses for each subscriber refined by the mobility profiles

Content








Sensor

Predictive Modeler

Scorer

Analytics Engine

High Velocity

High Volume

Real-time Adaptive Analytics

Adaptive Analytics

• Collaboration across tools

• SPSS and iLOG to manage models and rules

• PDA to do query processing for the models

• Streams to run the model

• PMML to flow models from SPSS / iLOG to Streams

Content








Marketing Assets

Resource Link

IBM Big Data Hub – Telco Home

Pagehttp://www-01.ibm.com/software/data/bigdata/industry-

telco.html

IBM Big Data Hub Cross-industry http://www.ibmbigdatahub.com/

Light Reading Webinar – “Big Data

dramatically changes the Telco

Game Plan”

http://www.lightreading.com/webinar.asp?webinar_id=300

92&webinar_promo=1000000332

Big Data Analytics (e-book) http://ibm.co/Zw0jRW

Big Data Analytics for

Communications Service

Providers (whitepaper)

http://bitly.com/RJHbhj

Telco Industry Blog on IBM Big

Data Hub (Author - Gaurav

Deshpande)

http://www.ibmbigdatahub.com/blog/author/gaurav-

deshpande

Videos http://www.youtube.com/watch?v=FIUFYyz03u8

http://www.youtube.com/watch?v=eg8KSLAZ2HM

http://pro.gigaom.com/webinars/netezza-making-big-

data-analytics-pay/

http://youtu.be/bdJu1Pt374g

http://www-01.ibm.com/software/data/bigdata/industry-telco.html

http://www.ibmbigdatahub.com/

http://www.lightreading.com/webinar.asp?webinar_id=30092&webinar_promo=1000000332

http://ibm.co/Zw0jRW

http://bitly.com/RJHbhj

http://www.ibmbigdatahub.com/blog/author/gaurav-deshpande



http://pro.gigaom.com/webinars/netezza-making-big-data-analytics-pay/

http://youtu.be/bdJu1Pt374g

IBM Big Data / Advanced Analytics Value Proposition

All Telco Data

Combine Network Data (usage, performance, capacity), Billing Call Detail Records, Subscriber, Channel, Policy, Device, Social etc.

At ScaleAbility to manage the stored Petabytes of data and incoming billions of records per day

At Speed of Business

Only IBM

Ability to process data and analytics in real time and close to point of origination to support emerging use cases such as Location Based Services (LBS) and Machine to Machine (M2M)

Only IBM can deliver the complete end to end technology and skills to capture quickly the new ERA value of Telco Big Data

Communities

• On-line communities, User Groups, Technical Forums, Blogs, Social networks, and more

o Find the community that interests you …

• Information Management bit.ly/InfoMgmtCommunity

• Business Analytics bit.ly/AnalyticsCommunity

• Enterprise Content Management bit.ly/ECMCommunity

• IBM Champions

o Recognizing individuals who have made the most outstanding contributions to Information Management, Business Analytics, and Enterprise Content Management communities

• ibm.com/champion

http://bit.ly/InfoMgmtCommunity

http://bit.ly/AnalyticsCommunity

http://bit.ly/ECMCommunity

http://www.ibm.com/software/data/champion

Thank YouYour feedback is important!

• Access the Conference Agenda Builder to complete your session surveys

o Any web or mobile browser at http://iod13surveys.com/surveys.html

o Any Agenda Builder kiosk onsite

http://iod13surveys.com/surveys.html

Technology

Big Data & Analytics Architecture