Embed Size (px)
Citation preview
Big Data VisualizationsA Dialog Surrounding the Use of Visualizations with Big Data
© Global IDs. All Rights Reserved.
Robert J. Abate, CBIP, CDMPOctober 18th, 2018
• Big Data Visualizations – A Starting Position
• Information Is The New Currency
• What Has Been Missing, The Challenge
• One Screen Is Not Enough
• Infographics And Big Data
• Best Practice Big Data Visualization Examples
• What Is Next?
Agenda
2
NOTE: Authors Opinion (not that of NY DAMA Chapter)…
Everyone Talks About It,
Few Really Know How To Do It,
It’s Been The Source Of Many Rumors,
Everyone Thinks That Everyone Else Is Doing It,
So Everyone Claims They Are Doing It !
3
Big Data Visualizations Can BeLikened To Teenage Relationships
Introduction To Big Data Visualizations
• Information is not data
• Data Science is almost the inverse of Business Intelligence (B.I.)
• Big Data Science has completely different goals - compared to Data Warehousing & B.I.
• Big Data architectures are unlike DW & BI architectures
• Examination of Big Data & Data Science requires new methods and thinking
Overview Of Big Data Vs. Business Intelligence…
4
• Real-time decision
making is critical to
business success
• Data in the enterprise
continues to grow
exponentially year
over year, making
analysis problematic
• To turn structured
and unstructured
data into actionable
intelligence, your
business needs an
effective, smart way
to harness Big Data.
• Agile / SCRUM
development is
essential today in Big
Data to learn
‘quickly’
Overview – Time Is Money & Lifecycle
5
Big Data Visualizations Defined
Authors Starting Opinion On Visualizations…
• One Screen Does Not Support “Ah-Ha” Moments1
• “Infographics” - are graphic, visual representations of information, data or knowledge2
– Intended to present complex information quickly/clearly
– They can improve cognition utilizing graphics to enhance human visual system’s ability to see patterns and trends
• Visualizations Provide A Picture Of Information
– Rapidly conveying insight & understanding when used right
1 – “The Work Of Edward Tufte & Graphics Press” www.edwardtufte.com2 – “Infographic” www./Wikipedia.com
6
A Picture Is Worth A Billion Words…
7
• Information Is Currency1
– It will be the most valuable asset in our organization
– Without information, nothing else we own will have value
– “Without knowledge, you’re powerless” 2
• Need For Trust– Legal compliance (SOX, …)
– Data integrity (Spreadmarts)
– Crypto-currency
“A testament to this new reality is the growing value of digital content. People download
songs, purchase movies online, exchange emails, and share personal information—all in
the form of digital content. Information in its many new forms has become
commoditized. In a digital world, ’information is the new currency’”3 - Aug 28, 2015
1 – http://www.pwc.com/gx/en/information-security-survey/pdf/safeguarding_the_new_currency.pdf2 – http://www.tieto.com/top-stories/information-strategic-business-asset-are-you-managing-it-such3 - https://blogs.opentext.com/why-information-is-the-new-currency/
Information is the New Currency
8
• Turning data into knowledge and insight utilizing metadata & context
• MDM / integration of assets
• Confidence and trust in data
• Linking people to information
• Making the data available
• Single Source Of Truth!
Business Architecture
Information Architecture
Application Architecture
Infrastructure Architecture
“Databases In Most OrganizationsAre Like Buildings Build Upon One-Another Without Thought Of Integration”
What Has Been Missing?
9
• Imagine Improving Productivity by
80%
What Are The Challenges Today
10
Toolset Capabilities:
• Discovery
• Profiling
• Classification
• Semantic Matching
• Global Domain Matching
• Security / PII Considerations
• Data Quality
• Data Governance
• Compliance (GDPR, Privacy)
• Data Analytics
Big Data Visualization Requirements:
• What data do we have
• How good is our data
• What types of data do I have
• Where are all of these types
• Logical groupings / MDM
• Don’t look at your neighbors data
• Let’s setup a means to monitor
• Let’s define people/tech/processes
• Let’s utilize this to manage data
• Let’ have all the semantic objects moved into a table that I can use
Global IDs - Data Ecosystem Tool Set
11
DISCOVERY
Structured Un-Structured
PROFILING
Global IDs Tool Set – Suite of Integrated Tools
12
CLASSIFICATION DOMAIN GROUPING
Global IDs Tool Set – Suite of Integrated Tools
13
DB Column Data QualitySemantic Object Governance Data SecurityData
RedundancyCompliance
Global IDs Tool Set – Purpose-Built Portals
14
DATA LINEAGE
Global IDs Tool Set – Viewing Lineage & Impact Analysis
15
Semantic Domains Identified In ColorProtected (PII/GDPR) Data
Visualizing a Data Universe
16
• Information was trapped by constraints
– Locked into silo’s of systems (dates)
– Projects created one-off solutions
• Data was everywhere – but not cataloged nor managed
– Tending to downplay quality, challenges and risks
• More than one dictionary in enterprise
– Item, Etc.
– Business Terms
• Brittle, tightly-coupled systems
Providing Insights – The Sam’s Club Challenge
17
Immersive Collaborative
Experience
Big Data & D/W
Integrated Fabric
Advanced Dynamic
Visualizations
People, Process &
Change Adoption
Objective:Create a comprehensive
repository of connected
data to enable one
version of the truth
across functional
boundaries leading to
better informed business
decisions
• Improve Productivity And Time To Finding Answers
• Visualization Of Information Provides Key Competitive Advantage
• Adding Value With “Big Data” Sources
Walmart Data CAFE
One Screen Is Not Enough
Combining Visualizations Improve Cognition
18
Infographics & Big Data - The Sam’s Club Solution
• Big Data Visualizations Are Not A Buzz Word Anymore
‒ Variety, Velocity, Volume & Veracity Of Data
Getting All The Sources Into The Data Lake Was Key
Used Extreme Agile Implementation (4 months)
‒ Visualizations “Freed The Information”
Providing Linkages Between Information Was Key
• Business Immediately Found Value
‒ Utilizing Pre-Built Visualizations With Touch-Screens
• Information Provided Insight For Data Science
19
Combined Visualizations Improve Cognition
Category
Sub Category
Fine Line
Fine
Lin
e 1
Fine
Lin
e 2
Fine
Lin
e 3
Fine
Lin
e 4
Fine
Lin
e 5
Fine
Lin
e 6
…
Best Practices in Big Data Visualizations
20
Measures and More Measures!
• Federal Reserve Chair Janet Yellen uses a dashboard of jobs data‒ In past, a single measure
(unemployment rate)‒ Make economic and labor policy
decisions• Instead, she uses dozen different
measures‒ Providing a more holistic, accurate,
and actionable view of the United States economy
• Single measure of anything complex – Poor!‒ Whether it’s the U.S. economy‒ Customer satisfaction and
predictive maintenance‒ Oversimplifying something to the
point of not being useful or actionable
More Best Practice Big Data Visualizations
21
Additional Best Practices in Big Data Visualizations
“Seeing” the Data Points Stretches the Mind…
22
Customer Views For Differential Analysis
Recency Frequency Value Tenure
Best Practice Data Visualizations Have Multiple Dimensions…
23
Showing Time Periods Geospatially
Further Best Practice Big Data Visualizations
24
Showing Movement Of Customers
Utilizing “Data Science” Groupings
Out of over 300 data elements, a total of 45 visit behavior features were found to be significant in predicting customer behavior
Further Big Data Visualization Best Practices
25
So What Is Next?
26
• Truly Integrated Data Environment – “Data Nirvana”
• Extreme Scale, Elastic Expansion• Semantic Integration – “All Visualizations & Reports Match”• Linearly Scalable
‒ Add 1x and get 2x performance
• Self – Service Provisioning‒ Users spin up a sandbox of data using movement tools with limited
duration
• Single Point Of Management‒ Resource utilization & optimization
• Secure, Unified Data Access – Single Point of Entry‒ Portal based sharing of data sandboxes (wiki-type)
• Reduce TCO By Eliminating Excessive Licensing Fees‒ Use of open source community to improve solution
The “Ideal” Big Data Visualization Ecosystem
27
• Big Data Visualizations Are The Future‒ You will not see other “reporting” as they don’t provide value
• Use More Than One “Screen or Concept” To Gain “Brain or Cranial” Insights
• Build With Best Practices‒ Build Iteratively – Data Sources & Graphics Integrations
‒ Combine Up To Four Measures Per Graphic
‒ Visualize Movement Pictorially
Customers Moving By Group
Geospatial Movement By Time
• Plan For Extreme (Web) Scale
Key Takeaways…
28
