Embed Size (px)
DESCRIPTION
We are living in a time of tremendous convergence, convergence of mobile, cloud and social…This convergence is forcing companies to change. At Nokia, we are changing the way we make decisions, from a manufacturing model to a data driven one. Yet making cultural changes is one of the hardest things to accomplish. In this talk, Amy O’Connor will highlight the journey Nokia is taking to evolve its culture - from building a platform for cultural evolution on top of Hadoop, to the administration of Nokia’s data, to how the company conducts the analysis that is enabling Nokia to compete with data.
Citation preview
Using Hadoop to
Change Company Culture
Amy O’Connor
Senior Director, Analytics
1
2
The Amazing Everyday
The Amazing Everyday
3
NOKIA’S HISTORY: 1865 TO Now
4
Data is our newest
raw material
5
US Government data
suggests worsening
conditions in urban
areas…this year
average commute time
has risen by 9 minutes.
TRAFFIC
Federal Highway
Commission, Urban
Congestion Report
January 2011 to
March 2011
6
Animate..? Something to just
make more visually appealing
Note deleted google map
TRAFFIC
IHS Automotive,
October 2011
Global car sales are
growing: there will
be 50% more cars
sold in 10 years as
there are today.
7
More data usually beats
better algorithms Anand Rajaramann
8
• Image Sensors
• Accelerometers
• Gyroscopes
• Compasses
• Pressure Sensors
• Microphones
• Light Sensors
• Assisted GPS
Probe points
collected
monthly from
Nokia alone
Devices in
use around
the world
9
10
80M
18M 24
Data Storage & Analysis Landscape
11
Data Silos
12
Traffic Probes
Ad Data
Search Logs
Places Registry
Consumer Profile
Device Data
Smart Data
Combining
sets of
behavioral &
contextual data
13
14
A Good Way to Change Corporate Culture
Getting Children to Eat Peas… Tell them you expect them to eat their peas.
Reward them with ice cream if they did.
Explain why it’s good for them to eat their peas.
Eat your own peas as a good role model.
Leann Lipps Birch, Head of Human Development & Family Studies Pennsylvania State University
15
Getting Children to Eat Peas…
Leann Lipps Birch, Head of Human Development & Family Studies Pennsylvania State University
16
Put them with children who love peas.
Change the stories they tell.
Nokia Data
Asset
Search
Log Files
Points of Interest
Advertising
Ad Inventory
Ad Canvas
Campaign Promotions Location
Premium Content
Favorite Routes
Log Files
Map Tiles
Imagery
Social
Social UGC
Universal Share
Journeys Data
Event Info
Navteq
All Probes Data
3D Imagery
Street View
Feature Recognition
Identity
Consumer Profile
SSO
Device Activation
Campaigns
Contacts
Device Programs
NAC, IIA
Windows Phone
Panel data
Equipment Master
Factory Data
Care & Marketing
Nokia IT
Registrations
Device Updates
Media
Products, Transactions
Songs, Delivery
Device/User CRM
Activation Info
Net Promotor
PC Suite
17
18
Users
Domain Expertise, Computer Science
Developers/ Applications
Domain Expertise, Statistical Skills,
Computer Science
Data Scientists
Domain Expertise, Statistical Skills
Data Analysts
Domain Expertise
Decision- Makers
Map Reduce
Pig Hive Oozie
TeraData Key Value
Store
Predictive Analytics
Dashboards Offline
Analysis
HBase
HBase
HBase
FTP
Scribe
Flume
HDFS
Analytics
19
Collaborative Working Model
Load
Transform
Extract
Platform
Present
Analyze and
Aggregate
Map Reduce
Pig Hive Oozie
TeraData Key Value
Store
Predictive Analytics
Dashboards Offline
Analysis
HBase
HBase
HBase
FTP
Scribe
Flume
HDFS
Analytics
20
Load
Transform
Extract
Platform
Present
Analyze and
Aggregate
Map Reduce
Pig Hive Oozie
TeraData Key Value
Store
Predictive Analytics
Dashboards Offline
Analysis
HBase
HBase
HBase
FTP
Scribe
Flume
HDFS
Analytics
Collaborative Working Model
Co-located developer clusters, pre-production cluster, product cluster
Create Hive
Schema
User Metadata Interfaces
Metadata Catalog
Data Model
Definition
Develop Validation
Job
Develop Cleanse
Job
Monitor and
Manage Transform
Create Library of MR Jobs
Develop Partition
Job
Catalog Data
Sources
Define Custom
ETL
Define Standard
ETL
Monitor and
Manage Data Feed
Integrate Streaming
Data
Integrate Historical
Data
To DataOS,
Structured Data
BI Tools: SPSS,
Tableau, Cognos
Customer Dashboard
AD Hoc Reporting
Hive QL and Pig Queries
MR Agg Job and Data to Oracle
MR Analysis
Job
Rec, Engines
Mahout Machine Learning
Data OS Product Teams and/or DataOS
MR Agg Job and Data to
Teradata
21
Smart Data: Behavioral & Contextual
Map Reduce
Pig Hive Oozie
TeraData Key Value
Store
Predictive Analytics
Dashboards Offline
Analysis
HBase
HBase
HBase
FTP
Scribe
Flume
HDFS
Analytics
Local Search
NAC
Ad Router
Standard ETL
• Clean data, remove bad record and health checks
• Partition data by date, hour, type
• Archive raw data
Aggregation
• Merge multiple data sources
• Implement app logic (e.g.; round up latitude/longitude to 3 decimals)
Aggregation
• Update top searches table and geo activity table in Oracle
22
Merchant Portal Heatmap
23
Mapping the World
© 2011 Nokia Company Confidential
23
24
Mapping the World
© 2011 Nokia Company Confidential
Probe data indicates the location, speed, heading, time etc. about a mobile device.
Billions of probe records per week.
24
Covers almost the entire world.
25
100% AGR
Probe Density: Urban and Arterial
26
792527719 (Jackson Blvd/Financial Pl)
From Ref, One lane
27
24 Hours in Our Analytics Ecosystem
~2TB ingested
350M messages via scribe
>3000 MR jobs
10TB processed
28
Spreading the Word
Smart Data Newsletter Stories
Realtime Dashboards
Data Asset Catalog
29
Thanks!
30
The Amazing Everyday
