Upload
ca-technologies
View
566
Download
0
Tags:
Embed Size (px)
Citation preview
Big Data Big Picture:Can You See It?
Mike HarerSenior Principal Product Manager, Big Data
Troy Coleman,Senior Principal Product Manager, Chorus and DB2 for z/OS
2 © 2015 CA. ALL RIGHTS RESERVED.
Today’s Discussion
THE FUTURE
EMERGING NEEDS
BIG DATA MANAGEMENT INNOVATIONS FROM CA
THE PATH FORWARD
1
2
3
4
Q & A5
The Future
4 © 2015 CA. ALL RIGHTS RESERVED.
THIS IS THE AGE OF THE APPLICATION
ECONOMY
AND IT’S ALL ABOUT THE DATA…
DATA IS THE FUELTHAT CAN MAKE OR
BREAK YOUR BUSINESS
5 © 2015 CA. ALL RIGHTS RESERVED.
DEMANDING GROWTH
SPINNING UPNEW PROJECTS
USING ALL DATA AVAILABLE
Businesses Moving Fast Into Big Data
Source: 2015 CA Sponsored Research: Vanson Bourne Global Big Data User Survey
60% want better customer experience
54% want new customers
84% will deploy an initiative in the next 1
year
94% plan to use all data available to them
6 © 2015 CA. ALL RIGHTS RESERVED.
How The BI Ecosystem Is EvolvingExample: Retail Store Analytics
NAS
POST-2014
HADOOP DISTRIBUTED FILE SYSTEM(UNSTRUCTURED/STRUCTURED)
NFSFTPETL
M/RHivePig
RMahout
RDBMSHBASE
StormKafka
CassandraDrill
SolrElastic Search
SAN(RDBMS/FILES)
Prod. CatalogTransaction Log
(Credit)Activity Log
Log AnalysisCustomer Insight
Recommendation Engine
Web Session AnalysisTransaction AnalysisPersonalized Offers
Transaction CaptureWeb Sessions
Shopping Carts
FLASH
INGESTION BATCH PROCESSING/MACHINE LEARNING
OLTP STREAMPROCESSING
INTERACTIVEANALYTICS
SEARCH
PRE-2014
Extract, Transform & Load all data into
DatawarehouseCapture All Transactions in
ERP/Relational DBPublish Promotions every
weekCreate Datamarts and Run Reports every day/week
ERP OPERATIONSOLTP OPERATIONS
ETL OPERATIONS BUSINESS INTELLIGENCE
BATCH OUTPUT MANAGEMENT
SAP, OraclePeoplesoft
InformaticaFTP/SFTP
Oracle DB, DB2DBSQL Server
CRON, Email
7 © 2015 CA. ALL RIGHTS RESERVED.
Result Is Complex Ecosystems
MULTIPLE → DOMAINS – IMPLEMENTATIONS – ENVIRONMENTS
Big Data Analytics Platforms
Traditional BI –Data Warehouse
Mainframe and Distributed
Amazon EMR Console
Unstructured Data Structured Data
8 © 2015 CA. ALL RIGHTS RESERVED.
Key Insight: Complexity Is The New Normal
GROWTH IN # OF BIG DATA PROJECTS OVER TIME
AS BIG DATA DISRUPTS COMPUTING PARADIGMS – GET AHEAD OF THE MANAGEMENT OF INFRASTRUCTURE NOW OR FACE THE
CHALLENGES OF DEALING WITH COMPLEXITY
Emerging Needs
10 © 2015 CA. ALL RIGHTS RESERVED.
What We Hear From Users
Source: 2015 CA Sponsored Research: Vanson Bourne Global Big Data User Survey
92%admit to major challenges managing their organization’s infrastructure to support current and future Big Data initiatives
31%
36%
40%
43%
48%
Lack of visibility into information andprocesses
Difficulty reintegrating analysis
Cost
Complexity of managing so many differentsolutions
Complexity of managing such a largeimplementation
COMPLEXITY IS 2 OF THE 5 LARGEST CHALLENGES IN MANAGING BIG DATA ENVIRONMENTS
11 © 2015 CA. ALL RIGHTS RESERVED.
Key Project Success Factors
Source: 2015 CA Sponsored Research: Vanson Bourne Global Big Data User Survey
40%
45%
47%
49%
57%
Cloud/hosted infrastructure services
Management tools for the infrastructure in place
Hire new resources with required skills
New infrastructure (storage, etc.)
Train existing resources on Big Data technologies
TOP 5 MAJOR INVESTMENTS NEEDED FOR PROJECT SUCCESS
12 © 2015 CA. ALL RIGHTS RESERVED.
The “Big” Big Data Management PainsThe Need to Overcome Many Challenges
Managing complex multi-vendor Big Data environments
Finding Hadoop/Big Data experts
Understanding capacity requirements for rapidly changing business needs
As complexity increases, manual processes are often required
System problems are hard to isolate, downtime increases
Unique tools and shortcomings
Driving forces… acquisitions, department consolidations demand greater operational efficiency
Big Data Management Innovations from CA
UNLOCK DATA
Getting broader: Unlocking insights from your mainframe data
VSTORM CONNECT DATA STREAMING FOR BIG DATA
MANAGE INFRASTRUCTURE
Getting to growth: Big data projects are critical for business value –protect this revenue by managing diversityCA BIG DATA
INFRASTRUCTURE MANAGEMENT
BDIM Pre-Release Registration and to Learn More:
www.ca.com/bigdata
14 © 2015 CA. ALL RIGHTS RESERVED.
DESIGNING FOR ROLE OF BIG DATA ADMINISTRATOR
The Role Of The Big Data Administrator
Perform day-to-day operations and support of Hadoop infrastructure
Monitor/maintain existing clusters and provision new ones
Integrate enterprise monitoring tools
Analyze current workloads and perform capacity planning
Hadoop Multi-Vendor Management
Hadoop Resource Management /Reporting
Hadoop Process Management / Automation
Hadoop Job Management & Monitoring
Hadoop System Health Monitoring & Alerts
ROLE / RESPONSIBILITIES: KEY MANAGEMENT CAPABILITIES:
15 © 2015 CA. ALL RIGHTS RESERVED.
360 Degree Approach
SINGLE, CONSISTENT MANAGEMENT UI EXPERIENCE
Linux / x86
SINGLE ACCESS POINT INTO
HETEROGENEOUS ENVIRONMENT
OPERATIONALIZE , MANAGE MULTI-VENDOR HADOOP MANAGEMENT DOMAINS
Big Data Infrastructure Management Server
CA BIG DATA INFRASTRUCTURE MANAGEMENT BIG DATA INFRASTRUCTURE
Current
Future Mainframe
HA
DO
OP
ECO
SYSTEM
16 © 2015 CA. ALL RIGHTS RESERVED.
CA Big Data Infrastructure ManagementSingle Unified View
Job monitoring
Heterogeneous system management
Intelligent alertmanagement
Resourcereporting
Cluster/Job/Node management
BDIMPre-Release Registration
and to Learn More: www.ca.com/bigdata
17 © 2015 CA. ALL RIGHTS RESERVED.
vStorm Connect Data Streaming for Big Data
It’s extremely difficult for Data Scientists, CMOs and other stakeholders to get access to their raw System z data in tandem with machine logs and other types of transactional information.
Need to make it easier for Mainframe customers to participate in Big Data projects. Simplifies and increases the ability to move Mainframe data into the Hadoop Big Data environment.
LAS VEGAS, November 10, 2014 — CA WORLD ’14 — CA Technologies (NASDAQ:CA) today announced a new global distribution agreement with Veristorm, a software company focused on Big Data management. The agreement strengthens CA’s ability to help customers leverage key business data on the mainframe for Big Data and analytics projects.
Logs
IMS
VSAM
Datacom
IDMS
DB2
WHY?
vStorm Connect Data Streaming for Big Data A new software solution that enables efficient data
integration into the Big Data ecosystem. Provides secure, near real-time access to mainframe
and distributed data for processing and self-service analytics.
New Extract-Hadoop-Transform technology, much faster than traditional ETL and staging processes, allowing enterprise data to be accessed by the analytics platform of choice.
WHAT?
18 © 2015 CA. ALL RIGHTS RESERVED.
vStorm Connect Data Streaming for Big DataHow it Works
vStorm Connect extracts data from DB2, VSAM, IMS, IDMS and Datacom and streams into Hadoop on the customer platform of choice (System z, Power Series or x86) thru a point-and-click, self-service portal.
Decreases the use of Mainframe general processors (reduction in MIPS/MSU utilization and associated expenses).
Lowers Mainframe storage costs, which can often reach up to $100k per TB, by offloading System z data into Hadoop off-platform.
Enables customer to gain new insight into key factors driving business performance.
Superior Technology - Unique because it does not require staging like traditional ETL solutions, lowering TCO & providing the data scientist real-time access to their most important transactional data.
HOW IT WORKS
KEY BENEFITS
19 © 2015 CA. ALL RIGHTS RESERVED.
Obstacles to Include Mainframe Data
1 2
Data Governance as the data moves off z/OS operational systems
Data Ingestion from z/OS into Hadoop (on or off platform) is a bottleneck (MIPS & ETL cost, Security around data access and transfer, Process Agility)
Existing security policies must be applied to data access and transfer.
There needs to be high speed / optimized connectors between traditional z/OS LPARs and the Hadoop clusters
Ability to serve data transparently into Hadoop clusters on mainframe AND on distributed platform
LEAD TO KEY REQUIREMENTS:
20 © 2015 CA. ALL RIGHTS RESERVED.
Data Ingestion Challenges
Extract from proprietary formats.
Aggregate or summarize.
Staging
Transform
Load
Hadoop, MongoDB, Cassandra, Cloud, Big Data
Ecosystem, Java, Python, C++, Interface skills
JCL, DB2, HFS, VSAM, IMS, OMVS, COBOL Copybooks, EBCDIC, Packed Decimal, Byte ordering, IPCS, z/VM,
Linux on z
z/OS
Logs
IMS
VSAM
Datacom
IDMS
DB2
21 © 2015 CA. ALL RIGHTS RESERVED.
vStorm Connect – Mainframe data to MainstreamBroad Platform Coverage
ELT alternative: Extract-Hadoop-Transform saves MIPS, staging storage, complexity
Dest: Cloudera, Hortonworks, etc
Metadata preserved: Use SQL on Hive
Governance, sandbox: zDoop
Datacom
IDMS
Logs
IMS
VSAM
DB2
System Z
z/OS
vStorm Connect Data Streaming for Big Data
Data +
Metad
ata
No Staging!
vStorm Enterprise
Power SystemsvStorm Enterprise
x86vStorm Enterprise
Linux on z
Linux
22 © 2015 CA. ALL RIGHTS RESERVED.
CA’s Big Data Management Approach
Deliver analytics solutions that remove the need for special skills and knowledge of the tools and infrastructure
Enable use of MF data (structured and unstructured) for analytics on any platform
1
Manage the complex analytics environment and simplify the complexity
2
Secure and enable compliant access control to data in analytics databases
3
4
CA DB2 Database Management for DB2 for z/OS
24 © 2015 CA. ALL RIGHTS RESERVED.
CA Chorus for DB2 Database Management Modernizing Management of DB2 for z/OSReleased 4.0 in April 2015
Custom Dashboard
Performance WarehouseCharting and Comparing
Policy Health Alerts and Investigation
25 © 2015 CA. ALL RIGHTS RESERVED.
CA Database Management Solutions for DB2 for z/OSAgile Incremental Release 19.0
CA Database Administration
Suite for DB2 for z/OS
CA SQL Performance Suite for
DB2 for z/OS
CA Sysview Performance
Management Option for DB2
for z/OS
CA Database Backup and
Recovery Suite for DB2 for z/OS
Sprint Reviewswith Users
Prototyping
User Research
User Feedback
Usability Testing
ProductBacklog
4 week sprints
26 © 2015 CA. ALL RIGHTS RESERVED.
BDIMPre-Release Registration and to Learn More:
www.ca.com/bigdata
27 © 2015 CA. ALL RIGHTS RESERVED.
For Informational Purposes Only
© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. This presentation is intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer references relate to customer's specific use and experience of CA products and solutions so actual results may vary.
Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of March 01, 2015 and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.
Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product releasereferenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any contract.
CA does not provide legal advice. Neither this document nor any CA software product referenced herein shall serve as a substitute for your compliance with any laws (including but not limited to any act, statute, regulation, rule, directive, policy, standard, guideline, measure, requirement, administrative order, executive order, etc. (collectively, “Laws”)) referenced in this document. You should consult with competent legal counsel regarding any Laws referenced herein.
Terms Of This Presentation