BIG DATA, BIG ANALYTICS

EMERGING BUSINESS INTELLIGENCE

AND ANALYTIC TRENDS

FOR TODAY’S BUSINESSES

BY MICHAEL MINELLI, MICHELE

CHAMBERS, AND AMBIGA DHIRAJ

Contents

Figure 1.1 3

Figure 1.2 4

Table 1.1 5

Figure 2.1 6

Figure 2.2 7

Figure 2.3 8

Figure 2.4 9

Data/Compute Cluster Visual 10

Figure 3.1 11

Figure 3.2 12

Sample ASCII Code 13

Table 3.1 14

Figure 5.1 15

Figure 5.2 16

Figure 5.3 17

Figure 5.4 18

Figure 5.5 19

Figure 5.6 20

Figure 5.7 21

Figure 5.8 22

Figure 5.9 23

Figure 5.10 24

Figure 6.1 25

Figure 6.2 26

Figure 6.3 27

Figure 6.4 28

Table 6.1 29

Figure 6.5 30

Figure 6.6 31

Table 7.1 32

Table 7.2 33

Figure 7.1 34

• ERP1990s

• eCommerce2010s

• CRM • Big DataAnalytics

1980s 2000s

Figure 1.1 Timeline of Recent Technology Developments

c01.indd 6c01.indd 6 14/12/12 5:51 PM14/12/12 5:51 PM

3

SQLAnalytics

DescriptiveAnalytics

Data MiningPredictiveAnalytics

Simulation Optimization

• Count• Mean• OLAP

• Univariatedistribution

• Centraltendency

• Dispersion

• Association rules

• Clustering• Featureextraction

• Classification• Regression• Forecasting• Spatial• Machine

learning• Text analytics

• Monte Carlo• Agent-based

modeling• Discrete eventmodeling

• Linearoptimization

• Non-linearoptimization

BusinessIntelligence

Advanced Analytics

Figure 1.2 Analytics Spectrum

c01.indd 14c01.indd 14 14/12/12 5:51 PM14/12/12 5:51 PM

4

Table 1.1 Big Data Business Models

Improve Operational Effi ciencies Increase Revenues

Achieve Competitive Differentiation

Reduce risks and costs Sell to microtrends Offer new services

Save time Enable self service Seize market share

Lower complexity Improve customer experience

Incubate new ventures

Enable self service Detect fraud

Source: Brett Sheppard, “Putting Big Data to Work: Opportunities for Enterprises,” GigaOm Pro, March 2011.

c01.indd 15c01.indd 15 14/12/12 5:51 PM14/12/12 5:51 PM

5

conversion

re-p

erm

ission

win

back

stic

kiness

repu

rchaseconversion

acquisition relationshipconversion

permissioncapture

welcome

reviews

friend tofriend

browse

abandon

replenishment

transactional

cross sell

alerts

winback

re-permission

The cross-channel lifecycle marketing approach

conversionsegmented

email / mobile / social / display / web

permission

Figure 2.1 New School of Marketing Source: Responsys Inc. 2012.

c02.indd 29c02.indd 29 14/12/12 5:52 PM14/12/12 5:52 PM

6

ElasticSearchIndexer

PercolatorQuery

Near Real-Time Event Processing Framework

Innovative near-real time event processing framework can be used in multiple applications, which improves processing speed

Other potential data sources

• Customer profile• Transactions data• Customer emails• Interactions with call centers• Social media information• Customer history• 360 degree view of customer

Listeners

Flagging of SuspiciousActivities

CapgeminiFraud Detection

Algorithm

TransactionsDatabase

Figure 2.2 Fraud Detection Powered by Near Real-Time Event Processing Framework Source: © 2012 Capgemini.

c02.indd 36c02.indd 36 14/12/12 5:52 PM14/12/12 5:52 PM

7

CustomerAcquisitionCollections

ManagementInformation

Regu

lato

ry

Regul

ator

y

Regulatory

RegulatoryAccount

Management

Planning

Figure 2.3 Credit Risk Framework Source: Ori Peled.

c02.indd 40c02.indd 40 14/12/12 5:52 PM14/12/12 5:52 PM

8

Clinical Data

Health CareProvider Data

TransactionalData

EMR/EHR

Claims/ Prescriptions

Compliance/Sunshine Act

TreatmentPatterns

Referrals

Patient Registries

Standardsof Care

RWE

Figure 2.4 Data in the World of Health Care Source: Jim Golden, Accenture, September 2012.

c02.indd 45c02.indd 45 14/12/12 5:52 PM14/12/12 5:52 PM

9

datadata data data data

DFS Block 1

DFS Block 2

DFS Block 1

DFS Block 1

DFS Block 2

DFS Block 2

DFS Block 3DFS Block 3

Map

Map Reduce

Map

Compute Cluster

Resultsdata data data datadata data data datadata data data datadata data data datadata data data datadata data data datadata data data datadata data data datadata data data data

Datadatadata data data datadatadata data data data

datadata data data datadatadata data data datadatadata data data data

datadata data data datadatadata data data datadatadata data data data

datadata data data datadatadata data data datadatadata data data data

Source: Apache Software Foundation.

c03.indd 63c03.indd 63 14/12/12 5:54 PM14/12/12 5:54 PM

10

- -Inter Firewall Inter Firewall

Organizations will need to complement just intra-firewall insights withinter- and trans-firewall analytics

HighInformation-

to-Noise Ratio

Intra-Firewall Intra-Firewall Intra-Firewall Intra-Firewall Intra-Firewall

Collaboration of INSIGHTS—NOT DATA

Social DataLocation

DataLow

Information-to-Noise

Ratio

Trans-Firewall

RetailSpendData

WEALTH of DATA outside the FIREWALL

Inter-Firewall

WebTrendsData

Figure 3.1 Inter- and Trans-Firewall Analytics Source: Mu Sigma and author Ambiga is a cofounder.

c03.indd 78c03.indd 78 14/12/12 5:54 PM14/12/12 5:54 PM

11

Disruptive value and efficiencies can be extracted by cooperating andexploring outside the boundaries of the firewall

Inter-Firewall

Value Chain

Health Insurance + Pharmacy + Drug Maker

– Customer health care insights – How doesthe consumer value his options?

Outside the Value Chain

Search Engine + Retailer

– Behavioral insights and outcome – Howdid the customer choose what they finallybought?

New data explains previously unsolvableproblems

Consumer Social Interaction

– Social feed data (outside firewall) +clickstream data (within firewall)

Customer Price Elasticity

– Price tests data (within firewall) +competitive prices (outside data)

– What is the sensitivity to price changes inthe presence of competitor pricing?

Trans-Firewall

Enables significant breakthroughsbased on synergies in insights

Internal data is no longer a strongdifferentiator/game changer

Large volumes of data outside the firewall

Figure 3.2 Value Chain for Inter-Firewall and Trans-Firewall Analytics Source: Mu Sigma.

c03.indd 79c03.indd 79 14/12/12 5:54 PM14/12/12 5:54 PM

12

Sample ASCII Code: character to binary code

0 5 0011 0000 A 5 0100 0001

1 5 0011 0001 B 5 0100 0001

2 5 0011 0010 C 5 0100 0001

3 5 0011 0011 D 5 0100 0001

4 5 0011 0100 E 5 0100 0001

5 5 0011 0101 F 5 0100 0001

6 5 0011 0110 G 5 0100 0001

7 5 0011 0111 H 5 0100 0001

8 5 0011 1000 I 5 0100 0001

9 5 0011 1001 J 5 0100 0001

c03.indd 86c03.indd 86 14/12/12 5:54 PM14/12/12 5:54 PM

13

Table 3.1 How Data Is Measured

Unit of Measure Approximate Size Mathematical Representation Examples

KB 5 kilobyte 1,000 (10 3 or one thousand) bytes 2 10 or 1024 bytes A typical joke 5 1KB

MB 5 megabyte 1,000,000 (10 6 or one million) bytes 2 20 or 1,048,576 bytes Complete work of Shakespeare 5 5MB

GB 5 gigabyte 1,000,000,000 (10 9 or one billion) bytes 2 30 or 1,073,741,824 bytes Ten yards of books on a shelf 5 1GB

TB 5 terabyte 1,000,000,000,000 (or 10 12 ) 2 40 or 1,099,511,627,776 bytes All the X-rays for a large hospital 5 1TB Tweets; created daily 5 121TB; U.S. Library of Congress 5 235TB

PB 5 petabyte 1,000,000,000,000,000 (or 10 15 ) 2 50 or 1,125,899,906,842,624 bytes All U.S. academic research libraries 5 2PB Data processed in a day by Google 5 24PB

EB 5 exabyte 1,000,000,000,000,000,000 (or 10 18 ) 2 60 or 1,152,921,504,606,846,976 bytes Total data created in 2006 5 161EB

ZB 5 zettabyte 1,000,000,000,000,000,000,000 (or 10 21 ) 2 70 or 1,180,591,620,717,411,303,424 bytes Total amount of global data expected to be 2.7 ZB by end of 2012

YB 5 yottabyte 1,000,000,000,000,000,000,000,000 (or 10 24 ) 2 80 or 1,208,925,819,614,629,174,706,176 Today, to save all those bytes you need a data center as big as the state of Delaware

Source: “Total Amount of Global Data 2.7 Zettabytes,” idc.com , December 1, 2010.

c03.indd 87c03.indd 87 14/12/12 5:54 PM14/12/12 5:54 PM

14

Figure 5.1 Big Data Analytics

o What and when did ithappen?

o How much is impacted andhow often does it happen?

o What is the problem?

Statistics

o What is likely to happennext?

o What if these trendscontinue?

o What if?

Data MiningPredictive ModelingMachine LearningForecastingSimulation

o What is the best answer?o What is the best outcome

given uncertainty?o What are significantly

differing and betterchoices?

Constraint-based optimizationMultiobjective optimizationGlobal optimization

Descriptive Analytics(Business Intelligence)

Predictive Analytics Prescriptive Analytics

Information Management

c05.indd 100c05.indd 100 14/12/12 5:54 PM14/12/12 5:54 PM

15

Figure 5.2 Emerging Signal of Ocean Trash over Time Source: Ian Thomson, “Is the Ocean Safe from You and Your Boat?” MySailing.com , May 3, 2012, www.mysailing.com.au/news/is-the-ocean-safe-from-you-and-your-boat .

0

0.01

−60

−40

−20

0

20

40

60

80Day # 3650 Year # 10 70%

2 4 6 8 10

50 100 150 200 250 300 350 400

c05.indd 107c05.indd 107 14/12/12 5:54 PM14/12/12 5:54 PM

16

Figure 5.3 Potential Result of Ignoring Signals Source: iStockphoto.com.

c05.indd 108c05.indd 108 14/12/12 5:54 PM14/12/12 5:54 PM

17

Figure 5.4 Big Data Landscape Visualization Source: “AlphaVision Solutions—Marketplace Surveillance,” Aqumin , July 10, 2012, https://aqumin.fogbugz.com/default.asp?W44 .

c05.indd 112c05.indd 112 14/12/12 5:54 PM14/12/12 5:54 PM

18

Figure 5.5 Word Cloud Source: Creve Maples, “Beyond Visualization—Productivity, Complexity and Information Overload,” O ’Reilly Strata Conference, Event Horizon Corpora-tion, Santa Clara, California, February 3, 2011.

c05.indd 115c05.indd 115 14/12/12 5:54 PM14/12/12 5:54 PM

19

Figure 5.6 Revised Word Cloud Source: Creve Maples, “Beyond Visualization—Productivity, Complexity and Informa-tion Overload,” O ’Reilly Strata Conference, Event Horizon Corporation, Santa Clara, California, February 3, 2011.

c05.indd 116c05.indd 116 14/12/12 5:54 PM14/12/12 5:54 PM

20

Figure 5.7 Typical Interactive Dashboard Source: Visualization example from Tableau Software, www.tableau.com.

c05.indd 119c05.indd 119 14/12/12 5:54 PM14/12/12 5:54 PM

21

Figure 5.8 Interactive Location Visualization Source: “FLINKLABS Our Work,” July 10, 2012, www.fl inklabs.com/portfolio.php .

c05.indd 120c05.indd 120 14/12/12 5:54 PM14/12/12 5:54 PM

22

Figure 5.9 Visualization of Temperature Trends Globally Source: “FLINKLABS Our Work,” July 10, 2012, www.fl inklabs.com/portfolio.php .

23

Figure 5.10 Twitter Propagation across the Globe Source: JerThorp@blog.blprnt.com, “The Weight of Data,” TEDxVan couver, July 10, 2012, www.youtube.com/watch?v=Q9wcvFkWpsM .

c05.indd 121c05.indd 121 14/12/12 5:54 PM14/12/12 5:54 PM

24

Figure 6.1 Evolution of Data Science

Inform

ation Tech

nology

Consulting

Soft

war

e Pr

oduct

DecisionSciences

•M

ath

• B

usin

ess • Technology• B

ehavioral Sciences

c06.indd 129c06.indd 129 14/12/12 5:56 PM14/12/12 5:56 PM

25

Figure 6.2 Data Scientist Skills Source: Drew Conway.

Math & Statistics

KnowledgeHackin

g Sk

ills MachineLearning

DataScience

SubstantiveExpertise

Dange

r

Zone!

Traditional

Research

c06.indd 135c06.indd 135 14/12/12 5:56 PM14/12/12 5:56 PM

26

Figure 6.3 Categories of Analytics for Big Data

Descriptive AnalyticsWhat happened?

Inquisitive AnalyticsWhy did it happen?

Predictive AnalyticsWhat is likely to happen?

Prescriptive AnalyticsWhat should happen?

AdfE34d

$$$$

c06.indd 141c06.indd 141 14/12/12 5:56 PM14/12/12 5:56 PM

27

Figure 6.4 Communication Cycle

Create

CommunicateDevelop

CognitiveRepairs

CommunicationCycle

ImplementAlign AlignIncentives

Measure

c06.indd 142c06.indd 142 14/12/12 5:56 PM14/12/12 5:56 PM

28

Table 6.1 Professional Traits Required for Decision Sciences

Necessary Traits Description

Learning over knowing Ability to apply fi rst principles and structured approaches to problem solving as opposed to relying excessively on past domain expertise

Agility Ability to cope with continuous transformation

Scale and convergence Synergistic ecosystem of talent, capabilities, processes, customers, and partners that can be leveraged across verticals, domains, and geographies

Multidisciplinary talent Ability to apply business, math, technology, and behavioral sciences

Innovation Increase breadth and depth of problem solving by constantly researching and deploying emerging techniques, technologies, and applications

Cost effectiveness Ensure sustainability and institutionalization of problem solving across organizations

c06.indd 143c06.indd 143 14/12/12 5:56 PM14/12/12 5:56 PM

29

Organizations are limited by fear of collaboration and overreliance on proprietary information

Fear ofCollaboration

ProprietaryOrientation

Silo behavior

Data privacy concerns

Competitve fears

Skepticism

Lack of executive sponsorship

Closed culture

Not invented here (NIH)attitude

Illusion of independence

Unawareness

Limited innovation andconvergence

Inability to tap into the"wisdom of the crowds"

Duplication and wasted efforts

Lack of synergies throughcooperation

No opportunity for disruptiveinnovation

No cross-functional/cross-organization learning

Drivers

Costs

Figure 6.5 Fear of Collaboration

c06.indd 147c06.indd 147 14/12/12 5:56 PM14/12/12 5:56 PM

30

Figure 6.6 Analytic Team Models

Different forms of organizational structure will emerge best suitedto the analytical needs of the organization

Central shared services organizationserving multiple departments

Each department with its ownanalytics unit

Central coordination with localexecution

Description

Owner

Pros

Cons

C-level executive Business unit/functional heads C level executive with a team oflieutenants embedded withinthe business unitsRetains the agility

Access to the right breadth anddepth of skills neededAutonomy of functional unitsmaintained

Coordination and planning are easier said than done

Local/silo approachShort term focus

Not aligned with the business units

Risk of spreading too thin in areasthat need in depth focus

Lack of control and alignment tocorporate vision

Seen as a cost centerNot aligned with the business unitsSlow and Inflexible

Economies of scale in infrasturctureand processEase of promoting a corporate visionof analytics in service of strategiccapabilitiesCross functional collaboration

Centralized Decentralized Federated

c06.indd 148c06.indd 148 14/12/12 5:56 PM14/12/12 5:56 PM

31

Table 7.1 Privacy Landscape

Businesses ■ Increased need to leverage personally identifi able and sensitiveinformation for competitive advantage

■ Signifi cant investment in data sources and data analytics

Criminals ■ Dramatic surge in identity theft■ Sophisticated technology to exploit data security vulnerabilities

Consumers ■ Increased awareness and concern about collection, use, and disclosureof their personal information

Legislators ■ Responding to consumer concern by restricting access to and use ofpersonal information

■ Signifi cant impact and restriction for business

Source: Adapted from Andrew Reiskind.

c07.indd 152c07.indd 152 14/12/12 5:56 PM14/12/12 5:56 PM

32

Table 7.2 Types of Protected Information

Personally Identifi able Information (PII): any information that directly or indirectly identifi es a person

Sensitive Information: any information whose unauthorized disclosure could be embarrassing or detrimental to the individual

Other Information: any other nonidentifi able information about an individual when combined with PII

Name

Postal address

Email address

Telephone/mobile number

Social Security Number

Driver ’s license number

Bank/fi nancial account

Credit or debit card number

ZIP Code

Race/ethnicity

Political opinions

Religious/philosophical beliefs

Trade union membership

Health/medical information

Marital status/sexual life

Age

Gender

Criminal record

Preferences

Cookie ID

Static IP address

c07.indd 159c07.indd 159 14/12/12 5:56 PM14/12/12 5:56 PM

33

Figure 7.1 Privacy as Shareholder Value Source: Thomson Reuters.

Cost Avoidance& Reduction

LeveragingBehavioral

Data

EstablishingReputation &Legitimacy

Research& Compliance

Tomorrow

Today

ExternalInternal

c07.indd 166c07.indd 166 14/12/12 5:56 PM14/12/12 5:56 PM

34