Open source Big Data case study: Building a platform for ...€¦ · Open source Big Data case study: Building a platform for remote device support at NetApp (Part I – Business)

Open source Big Data case study: Building a

platform for remote device support at NetApp

(Part I – Business)

Copyright © 2012 Accenture All rights reserved. 2

Topics

Big Data Perspective

Case Study: NetApp AutoSupport

Observations from around the firm


Big Data

The concept is disruptive. The technology is disruptive. And, markets and

clients are being impacted.

1 Wordle for Credit Suisse, Does Size Matter Only?, September 2011


Shifts in Data and Analytics

Data-led Innovation

Data Explosion • Unstructured data is doubling

every 3 months

• 2011 saw 47% growth overall

• By 2015, number of networked

devices will be 2x global

population

• De-coupling data from

applications

• Disparate external data shaping

context

• Cost effective mobilization of

massive scale data

Monetization • Growth of enterprise data

monetization services

• Large retailers monetizing own

data to provide insights to

suppliers

Social Media • Growing market for scrubbed,

aggregate data from social

media and blogs

• Greater focus on data that

provides insight in a customer’s

digital persona

Technology

• Commodity priced storage and

compute

• Emergence of open source and

big data technologies solving

production problems at scale

Data Mobilization • Novel approaches to analyze

unstructured data creating

shorter time from data to insight

• Shift towards data consumption

in multiple environments

(business apps, mobile, social)

The changing landscape and required winning strategies are creating shifts

within Big Data collection and analytics


The Big Data Approach

Culture

• Data-driven decision making

• Experimentation and

continuous improvement with

academic rigor

• End-to-end ownership of

services

• Sharing of platform, tools and

code

Treat data as a strategic asset, seek to maximize it’s value to the organization

Invest in common services, data platforms and tools

Rapidly prototype, deliver, and measure value-added data services, evolve over time


Topics

Big Data at Accenture


Observations from around the Firm


Client Context

NetApp, Inc.

• Industry: Data storage, data management

• 77% Fortune 500 companies are customers

• Creator of Data ONTAP: industry leading storage OS


• Secure automated “call-home” service

• Catch issues before they become critical

• System monitoring and alerting

• RMA requests without customer action

• Faster incident management

AutoSupport

AutoSupport

Messages AutoSupport

Data Warehouse Storage Devices


Business Challenges

• Increase in response times /

lower availability for services

• Incoming data volume doubling

every 16 months

• Proliferation of ad hoc datamarts

and point solutions

• Unable to analyze full dataset in

timely fashion

File StorageASUP

Messages

SAP CRM

ODS

DSS

MyASUP STOReBI Analytics & MiningASUP Tools

PWillows

BI

Light

Parser

Core

Parser

Xterra DB

Parser

DW 2 Adhoc DB’s

PMBTA

Adhoc

Parsers

DRM DMHDDSAP CRMSTAGE

PNOW GEO

Transform

Presentation

Interface

Rules

Integrate

RulesRules

Rules

Stage

Extract

Source

Xterra

Parser Parser

Custom ETLCustom ETL Custom ETLCustom ETL

eB

I

DW 3

Java Interface Jasper

Stored Proc

Loader

RulesRules

Rules

Rest InterfaceCRM Module

Rules Module

Xterra File

Various

RulesRules

Rules

0

500

1000

1500

2000

2500

3000

3500

Jan-05 Jan-06 Jan-07 Jan-08 Jan-09 Jan-10 Jan-11 Jan-12 Jan-13 Jan-14 Jan-15 Jan-16

AutoSupport Flat-File Storage Requirement

Total Usage (tb)

Projected Total Usage (tb)

Doubles


Solution Design Goals

• Improve system response times

and data availability

• Expose common data services for

consumption across business units

• Standardize key business metrics

into common rules repository

• Lower operational costs as

ecosystem continues to scale

• Provide more granular analytical

capabilities

Improve data access and technology cost effectiveness and performance.


Role of Open Source

Platform is composed of open source technologies purpose-built for large-scale

storage, processing and analysis

1 Actual Big Data Solution Blueprint for a hybrid deployment


Outcomes

• Exposed full device details to end users for analysis

- New predictive maintenance capabilities, historical

trending

• Single unified architecture for analysis

• Improved system uptime and performance

• Lowered operational footprint, hardware / licensing costs

• Quicker response to new business requirements

memory mapped I/O base 0xfdb00000, size 0x10000

slot 0: BGE 10/100/1000 Ethernet Controller

e0a MAC Address: 00:a0:98:12:c8:34 (auto-1000t-fd-up)

e0b MAC Address: 00:a0:98:12:c8:35 (auto-unknown-down)

Device Type: BCM5715C

memory mapped I/O base 0xfe510000, size 0x10000




slot 0: NVRAM (NetApp NVRAM VII)

Revision: A4

Serial Number: 739124

Memory Size: 2048 MB

Battery1 Status: Battery sufficiently charged (3906 mV)

Charger1 Status: OFF

Running Firmware: 2 (4.7.600)

Cluster Interconnect Port 1: 4x copper

Cluster Interconnect Port 2: 4x copper


Sample Use Case – Risk Prognosis

AutoSupport

Messages

NetApp Storage Systems AutoSupport

Data

Warehouse

Risk Detection

& Automation

Engine Available to Customers,

Partners and Support

Personnel

Risk Monitoring,

Notification and Mitigation

Procedure via My

AutoSupport

Rules and Policies

Automated Risk Prognosis

Enhancing System Uptime and Building Customer Loyalty

NetApp Support Personnel

Innovation and Governance


Topics

Big Data at Accenture


Observations from around the Firm


Observations and Trends

Cost Effective

Mobilization of

Data for

Analysis



Simplified Illustrative

View

EDW ODS

DSS

Application Application Application

Aggregations /

Summary

DW

DW

ETL

Decoupling

Data from

Applications



SESSION VIEW

Containing Session Level Summary

Information by IP Address

TIMELINE VIEW

Containing Session Level Event

Sequence Information by IP Address

CDN QUALITY VIEW

Quality Values for HLS Streaming by IP

Address

Sources

• CST

• CDN

• Location

• Billing

• Term Reg

Size

• CST 50M Records/Day

• CDN 100M Records/Day

• Load 1.2M Records /Min

Hardware

• 2 12-Core Mac Towers

• 64 GB Ram / 16 TB Disk iGUIDE

Specific Analysis To Root Cause STB not

Found Issue

External

Data

Shaping

Context



NoSQL

Real Time Recommendation Engine

Cloud

http event (pixel, Ajax, API Call,…) http

Machine Learning Engine

NoSQL Master Event Storage

Real-time Queries

Customer Profile Associations

Item-based Associations

Clustering

Regressions

HTTP Event Capture

Context API Cache

Internal Event Replication

Real-time Queries

Customer Profiles

Item purchase

Visualizations

Internal

Systems

Consumers

Deriving

Unique Value

by Combining

Internal Data



Data Collection

Data Ingest

Data as a Platform Analytical Database

Data Science Search & Discovery

Real Time

Integration

F

D

C

A

Big Data

Driving

Innovation


Thank you

Jonathan Bender Consultant, Accenture Technology Labs

[email protected]

Documents

Open source Big Data case study: Building a platform for ...€¦ · Open source Big Data case study: Building a platform for remote device support at NetApp (Part I – Business)