Upload
contexti
View
361
Download
0
Embed Size (px)
DESCRIPTION
Big Data is moving from hype to reality for many organisations. The value proposition is clear and sponsorship is high, but how do organisations execute? Join Oracle and Contexti to discuss the typical journey of a big data project from concept to pilot to production. • Discuss our experience with a regional Telco • Common Use Cases across key verticals • Defining and prioritising use cases • The challenge of moving from Pilot to Production • Common Operating Models for Big Data • Funding a Big Data Capability going forward • Pilots - common mistakes; challenges; success criteria
Citation preview
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Big Data: From Pilot to Production
Vicky Falconer - Oracle
Grant Priestley - Contexti
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Program Agenda
1
2
3
4
Big Data Project Challenges
Typical Big Data Journey
Common Operating Models
Technology Considerations
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Big Data Project Challenges
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Experience with a regional Telco
• You don’t know what you don’t know…
• Build it and they will come
• Technology versus capability
• Clear definition of skill requirements
• Moving from Pilot to Production
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Additional challenges
• Who owns data?
• What to do in house and what to externalise
– Analytics
– Admin
– Development
– Engineering
• Operationalising insight
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Key Lessons
• Where to start?
• Culture
• Scope
• Building capabilities
• Technology
Right Questions
Right Use Cases
Right Business
Case
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Typical Big Data Journey
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Big Data Developments
Increasing use of infrastructure as a
service
Better understanding of the possibilities
offered by unstructured
data
Moving from historical batch computing to
real-time analytics
Wider awareness and a
more defined understanding of
Big Data
Wider variety of vendors offering
Big Data solutions
Less hype, more real use cases of
companies exploiting Big
Data
Maturity of Big Data tools
bringing them into the
mainstream
What changes we have noticed over the past 12 months with respects to Big Data that are most likely to impact on your organisation or on the market in general:
Increase in requests for platform as a
service
AdvancedLess Advanced 2015?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Where Value Lies for Most OrganisationsThe proliferation of Big Data Analytics applications and solutions has given rise to the need for a Big Data Platform that enable these initiatives to occur and support all use cases including Advanced Analytics, Internet of Things (IoT) and the Digital Enterprise. The Big Data PaaS accelerates organisation's projects by provisioning the initial platform and development environment, eliminating the need for hard-to-find Big Data skills and ultimately allows the enterprise to focus on strategic initiatives and IP creation rather than platform operations.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Big Data Use Cases
1
2
3
Customer Insights / Behavior
Data Warehouse Augmentation
Risk Analysis & Fraud Detection
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Customer Insight / Behavior
Challenges
Understand customer behavior through predictive analytics
Solution
Leverage Hadoop and ML techniques to build “population-based behavioral” clusters enabling personalised content to be served up in certain real-time sequences
Business Outcomes
• Increase in sales conversion• Online engagement is personalised, as it is in store
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Data Consumer(s)Data Source(s) Contexti Big Data Platform
Customer Insight / Behavior
Semi-StructuredData
StructuredData
Pre-computed Web Content &
Deals
Raw / EnrichedData Sets
(HDFS / MFS)
StreamingData
Acquisition
File-BasedData
Acquisition
RDBMS-Based Data
Acquisition
Data Ingestion
Deep Analytics &
Machine Learning
Streaming Capability Serving Capability
Batch Capability
Online Store
Customers
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Data Warehouse Augmentation
Challenges
Reduce latency between data generation and availability
Solution
Offload ETL processing to Hadoop platform and support the ingestion of multi-structured data sets
Business Outcomes
• Access of data reduced from T+1, T+2 to real-time / intra-day• Reduce cost of ETL processing• More time now spent on analysing data than data wrangling
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Data Warehouse Augmentation
Data Consumer(s)Data Source(s) Contexti Big Data Platform
Semi-StructuredData
UnstructuredData
Raw / EnrichedData Sets
(HDFS / MFS)
StreamingData
Acquisition
File-BasedData
Acquisition
RDBMS-Based Data
Acquisition
Data Ingestion Extract Load
Transform (ELT) and Data
Preparation Processes
Streaming Capability Serving Capability
Batch Capability
Users
StructuredData
Reporting, Search &
Query
RDBMS & MPP
Platforms
Pre-computed Views
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Risk Analysis & Fraud Detection
Challenges
Reduce incidents of fraud through more sophisticated detection and monitoring
Solution
Ingested structured and raw law data from multiple applications and combined data filtering from Pig/Hive with statistical modeling by R,while executing CEP on streams of data
Business Outcomes
• Implemention of real-time trigger based analytics that provides early detection of fraud
• “Schema on read” provided greater flexibility for analysis
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Contexti Big Data Platform
Risk Analysis & Fraud Detection
Data Consumer(s)Data Source(s)
Semi-StructuredData
UnstructuredData
Raw / EnrichedData Sets
(HDFS / MFS)
StreamingData
Acquisition
File-BasedData
Acquisition
RDBMS-Based Data
Acquisition
Data Ingestion
Streaming Capability Serving Capability
Batch Capability
Online Store
StructuredData
Data Access Provisioning
API
RDBMS & AnalyticsPlatforms
Raw Data(In-Memory)
CEP / Stream Analytics
Pre-computed Views
Real-Time Incremental
ViewsFraud Systems
Deep Analytics &
Machine Learning
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Common Operating Models
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Common Operating Models
Decentralised Centralised Federated
In a decentralized services model, each business or function has its own analytics group, which enables and encourages rapid decision-making and execution.
Pros:• Analytics needs aligned to business
functions• Close to business and customers needs
Cons:• Limited strategic view• Duplication, redundancies and inability
to standardise or leverage scale
The centralised shared-services model exists outside organizational divisions or functions, in some cases external to the organisation itself.
Pros:• Standardised processes and methods• Independent viewpoints shared across
the organisation
Cons:• Perception that group lacks functional
expertise• Ownership of IP when outsourced
The federated shared-services model is a centralized model that rolls under an existing function or business unit and serves the entire organization.
Pros:• Speed in execution & decision making• Pre-existing shared service processes
and structure
Cons:• Less transparent resource allocation• Focus on business function priorities
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Building a Best Practice Analytics Capability
Analytical Capability Techniques Questions
BasicProvide static, historical view of business performance drawn on basic scorecard and static reports
Query and drill down Where is the problem?
Ad hoc reporting How many? How often? Where?
Standard reporting What happened?
AnticipatoryCreates transparency into past and future drivers, using systems and processes to perform a range of descriptive analytics
Segmentation Analysis What are the unique drivers?
Statistical Analysis Why is this happening?
Sensitivity Analysis What if conditions change?
PredictiveRequires high-quality integrated data and complex mathematical capabilities and offers dynamic forward looking insights
Optimisation What is the best that can happen?
Simulation What would happen if …?
Predictive Modelling What could happen next?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Challenges to FaceHurdles between Pilot and Production
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Pilot vs Production Characteristics
Pilot Characteristics Production Characteristics
Funding Project-Based Project and BAU Funding
Number of Use Cases 2–3 use cases > 5 use cases
Insights Demonstrated Actionable / Operational
Service Level No / Loose SLA (project-based) Enforced SLAs, OLA
Big Data Capability • Batch• Serve
• Batch• Serve• Stream (advanced)
Resiliency / DR No Mandatory
Security Enabled Optional Mandatory
Scale 1 Rack, <5 data sources Multiple Racks, >10 data sources
Timing 3-6 months 6-9 months*
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Is Your Big Data Pilot Ready for Production?
Culture
• Allows for trial and error, ability to fail
• Understanding that data is an enterprise asset, benefit of being “data informed”
Structure & Skills
• Governance of data and its use
• Decision on what skills to acquire/develop/buy (analyst, dev, data scientist, ops, engineering)
• Funding Model (How will users/customers be charged?)
Integration
• Technologies in place to connect internal/external data, including unstructured data
• Integration of “actionable insights” into operational processes
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Technology Considerations
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Technology Considerations
• Business Strategy drives IT Strategy
– Information Architecture
• Future State Infrastructure
– Scale out and up
– Adding Big Data to existing infrastructure can be complex
• Analytics
– Embed in operational systems
• Integration insight into existing systems and processes
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Actionable
Events
Streaming Engine Data Reservoir Enterprise Data & Reporting
Discovery Lab
Actionable
Information
Actionable
Data Sets
InputEvents
Execution
Innovation
Discovery Output
Data
Conceptual View
StructuredEnterprise Data
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Big Data Strategy
BY INDUSTRY & LINE OF BUSINESS
BIG
DA
TA
AP
PLI
CA
TIO
NS
DISCOVERY
BU
SIN
ESS
AN
ALY
TIC
S
BUSINESS ANALYTICS
DATA RESERVOIR
BIG
DA
TAM
AN
AG
EMEN
T
DATA WAREHOUSE
SOU
RC
ES
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Big Data Management System
SOU
RC
ESDATA RESERVOIR DATA WAREHOUSE
Oracle Database
Oracle IndustryModels
Oracle Advanced Analytics
Oracle Spatial & Graph
Big Data Appliance
Apache Flume
OracleGoldenGate
Oracle Event Processing
Cloudera Hadoop
Oracle NoSQL
Oracle R Advanced Analytics for Hadoop
Oracle R Distribution
Oracle Database
In-Memory, Multi-tenant
Oracle Industry Models
Oracle Advanced Analytics
Oracle Spatial & Graph
Exadata
OracleGoldenGate
Oracle EventProcessing
Oracle DataIntegrator
Oracle Big DataConnectors
Oracle DataIntegrator
ORACLE BIG DATA SQL
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle R Enterprise Approach
Data and statistical analysis are stored and run in-database
Same R user experience & same R clients
Embed in operational systems
Complements Oracle Data Mining
ROpen Source
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Closing Remarks
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |my.oracle.com/go/bigdata
Summary
• Where to start?
• Culture
• Scope
– Data Ownership
– Data Governance
– IM Strategy
• Building capabilities
• Technology
Right Questions
Right Use Cases
Right Business
Case