View
3
Download
0
Category
Preview:
Citation preview
Kirk Borne @KirkDBorne
The Self-Driving Organization and Edge Analytics in a Smart IoT World
http://caanz.co.nz/industry-voice/whos-afraid-of-big-data/ https://www.linkedin.com/pulse/can-elephant-bigdata-dance-iot-internet-things-tune-lambba
Abstract
I will describe the concept of a self-driving
organization that learns, gains actionable insights,
discovers next-best move, innovates, and creates
value from streaming big data and edge analytics
via ubiquitous data sources in the IoT-enriched
world. I will also present an Analytics Roadmap
for the Smart IoT-enabled Cognitive Organization.
2
OUTLINE • First came Data, then Big Data, now Smart Data
• The Self-Driving Organization
• Steps to Discovery: Data Science & Analytics
• Case Study: Mars Rovers (metaphor)
• Analytics Roadmap for Self-Driving Orgs
3
OUTLINE • First came Data, then Big Data, now Smart Data
• The Self-Driving Organization
• Steps to Discovery: Data Science & Analytics
• Case Study: Mars Rovers (metaphor)
• Analytics Roadmap for Self-Driving Orgs
4
Ever since we first explored our world…
http://www.livescience.com/27663-seven-seas.html 5
…We have asked questions about everything around us.
https://jefflynchdev.wordpress.com/tag/adobe-photoshop-lightroom-3/page/5/ 6
So, we have collected evidence (data) to answer our questions,
which leads to more questions, which leads to more data collection,
which leads to more questions, which leads to BIG DATA!
y ~ 2 * x (linear growth)
y ~ 2 ^ x (exponential)
7
https://www.linkedin.com/pulse/exponential-growth-isnt-cool-combinatorial-tor-bair
Big Wow!
y ~ x! ≈ x ^ x → Combinatorial Growth! (all possible interconnections, linkages, and interactions)
Semantic, Meaning-filled Data:
• Ontologies (formal)
• Folksonomies (informal)
• Tagging / Annotation – Automated (Machine Learning)
– Crowdsourced
– “Breadcrumbs” (user trails)
Broad, Enriched Data:
• Linked Data (RDF)
– All of those combinations!
• Graph Databases
• Machine Learning
• Cognitive Analytics
• Context
• The 360o view
What makes your data Smart?
The Human Connectome Project: mapping and linking the major pathways in the brain. http://www.humanconnectomeproject.org/
8
Internet of Things (IoT): Deploying intelligence at the point of data collection!
(Machine Learning at the edge of the network = Edge Analytics!)
Internet of Everything
https://www.nsf.gov/news/news_images.jsp?cntn_id=122028
The Internet of Things (IoT) will be an interconnected universe of Dynamic Data-Driven Application Systems (dddas.org) =>
Combinatorial Explosive Growth of Smart Data!
9
https://www.linkedin.com/pulse/can-elephant-bigdata-dance-iot-internet-things-tune-lambba
• Smart Health
• Precision Medicine
• Precision Farming
• Personalized Financial Services
• Smart Organizations
• Predictive Maintenance
• Prescriptive Maintenance
• Smart Grid
• Smart Apps
• Predictive Retail
• Precision Marketing
• Smart Highways
• Precision Traffic
• Smart Cities
• Predictive Law Enforcement
• Personalized Learning
Smart => Predictive, Precision, Personalized!
Ubiquitous Smart Data in the IoT: Applications and Use Cases are everywhere…
… that demands dynamic action = Edge Analytics
10
OUTLINE • First came Data, then Big Data, now Smart Data
• The Self-Driving Organization
• Steps to Discovery: Data Science & Analytics
• Case Study: Mars Rovers (metaphor)
• Analytics Roadmap for Self-Driving Orgs
11
What is Self-Driving? … ~Autonomous! (e.g., cars, cities, deep space probes, organizations)
Defining Characteristics Data Analytics Maturity Organizational Posture
Sensing, Streaming
Responsive
Learning, Agile
Contextual, Optimizing
Deciding, Acting
Note: additional KEY CHARACTERISTICS = Secure, Compliant, “Embedded Reporting” = explosive growth of “NC2” data (“in situ” data) that delivers CONTEXT
12
Defining Characteristics Data Analytics Maturity Organizational Posture
Sensing, Streaming Descriptive
Responsive Diagnostic
Learning, Agile Predictive
Contextual, Optimizing Prescriptive
Deciding, Acting Cognitive
Note: additional KEY CHARACTERISTICS = Secure, Compliant, “Embedded Reporting” = explosive growth of “NC2” data (“in situ” data) that delivers CONTEXT
13
What is Self-Driving? … ~Autonomous! (e.g., cars, cities, deep space probes, organizations)
Defining Characteristics Data Analytics Maturity Organizational Posture
Sensing, Streaming Descriptive Passive
Responsive Diagnostic Reactive
Learning, Agile Predictive Proactive
Contextual, Optimizing Prescriptive Predictive Reactive
Deciding, Acting Cognitive Best action, Smart
Note: additional KEY CHARACTERISTICS = Secure, Compliant, “Embedded Reporting” = explosive growth of “NC2” data (“in situ” data) that delivers CONTEXT
14
What is Self-Driving? … ~Autonomous! (e.g., cars, cities, deep space probes, organizations)
OUTLINE • First came Data, then Big Data, now Hyper-Data
• The Self-Driving Organization
• Steps to Discovery: Data Science & Analytics
• Case Study: Mars Rovers (metaphor)
• Analytics Roadmap for Self-Driving Orgs
15
Some Quick Definitions • Statistics = the practice (and science) of collecting and analyzing
numerical data
• Machine Learning (ML) = mathematical algorithms that learn from experience (historical data)
• Data Mining = application of ML algorithms to data
• Artificial Intelligence = application of ML algorithms to robotics and machines
• Data Science = application of scientific method to discovery from data (including statistics, machine learning, and more: visual analytics, machine vision, computational modeling, semantics, graphs, network analysis, data indexing schemes,…).
• Analytics = the products of machine learning & data science.
• Statistics = the practice (and science) of collecting and analyzing numerical data.
• Machine Learning (ML) = mathematical algorithms that learn from experience (historical data).
• Data Mining = application of ML algorithms to data.
• Artificial Intelligence (AI) = application of ML algorithms to robotics and machines.
Source for graphic: http://blogs.sas.com/content/subconsciousmusings/2014/08/22/looking-backwards-looking-forwards-sas-data-mining-and-machine-learning/
16
1) Class Discovery: Finding new classes of objects (population segments), events, and behaviors. This includes: learning the rules that constrain the class boundaries.
2) Correlation (Predictive and Prescriptive Power) Discovery: Finding patterns and dependencies, which reveal new governing principles or behavioral patterns (the “customer DNA”).
3) Novelty (Surprise!) Discovery:
Finding new, rare, one-in-a-[million / billion / trillion] objects and events.
4) Association (or Link) Discovery: Finding unusual (improbable) co-occurring associations.
Machine Learning : Algorithms that learn from experience
17
(Graphic by S. G. Djorgovski, Caltech)
5 Levels of Analytics Maturity
1) Descriptive Analytics
– Hindsight (What happened?)
2) Diagnostic Analytics
– Oversight (real-time / What is
happening? Why did it happen?)
3) Predictive Analytics
– Foresight (What will happen?)
4) Prescriptive Analytics
– Insight (How can we optimize what happens?)
5) Cognitive Analytics – Right Sight (the 360 view , what is the right
question to ask for this set of data in this
context = Game of Jeopardy)
– Finds the right insight, the right action, the
right decision,… right now!
– Moves beyond simply providing answers, to
generating new questions and new
hypotheses, for your “next-best move”
18
PREDICTIVE
Analytics
Find a function (i.e., the model) f(d,t) that
predicts the value of some predictive
variable y = f(d,t) at a future time t, given
the set of conditions found in the training
data {d}.
=> Given {d}, find y.
PRESCRIPTIVE
Analytics
Find the conditions {d’} that will produce a
prescribed (desired, optimum) value y at a
future time t, using the previously learned
conditional dependencies among the
variables in the predictive function f(d,t).
=> Given y, find {d’}.
19
PREDICTIVE
Analytics
Find a function (i.e., the model) f(d,t) that
predicts the value of some predictive
variable y = f(d,t) at a future time t, given
the set of conditions found in the training
data {d}.
=> Given {d}, find y.
PRESCRIPTIVE
Analytics
Find the conditions {d’} that will produce a
prescribed (desired, optimum) value y at a
future time t, using the previously learned
conditional dependencies among the
variables in the predictive function f(d,t).
=> Given y, find {d’}.
Confucius says…
“Study your past to know
your future”
20
PREDICTIVE
Analytics
Find a function (i.e., the model) f(d,t) that
predicts the value of some predictive
variable y = f(d,t) at a future time t, given
the set of conditions found in the training
data {d}.
=> Given {d}, find y.
PRESCRIPTIVE
Analytics
Find the conditions {d’} that will produce a
prescribed (desired, optimum) value y at a
future time t, using the previously learned
conditional dependencies among the
variables in the predictive function f(d,t).
=> Given y, find {d’}.
Confucius says…
“Study your past to know
your future”
Baseball philosopher Yogi Berra says…
“The future ain’t what it
used to be.”
21
Context is King! “It’s not what you look at that matters – it’s what you see.” – Henry David Thoreau
“You can see a lot just by looking.” – Yogi Berra
• Context is “other data” about your data = i.e., Metadata!
• The 3 most important things in your data are: Metadata, Metadata,
Metadata!
• Metadata are…
– Other Data that describes Other Data
– Other Data that describes Your Data
– Your Data that describes Other Data
• e.g., Connected “Smart” Cars = that car that is braking 3 vehicles
ahead of you = informs your vehicle to brake now!
• The Smart Business = predictive maintenance algorithm alerts the
corresponding asset, the right skilled technician, & the right tool to
converge at right place at the right time! 22
Levels of Maturity in the Self-Driving Organization
Note: additional KEY CHARACTERISTICS = Secure, Compliant, “Embedded Reporting” = explosive growth of “NC2” data (“in situ” data) that delivers CONTEXT
Defining Characteristics Data Analytics Maturity Organizational Posture
Sensing, Streaming Descriptive Passive
Responsive Diagnostic Reactive
Learning, Agile Predictive Proactive
Contextual, Optimizing Prescriptive Predictive Reactive
Deciding, Acting Cognitive Best action, Smart
23
OUTLINE • First came Data, then Big Data, now Hyper-Data
• The Self-Driving Organization
• Steps to Discovery: Data Science & Analytics
• Case Study: Mars Rovers (metaphor)
• Analytics Roadmap for Self-Driving Orgs
24
Case
Study:
Mars
Rovers (metaphor)
25
Mars Rover: intelligent data-gatherer, mobile data mining
agent, autonomous decision-support system, and self-driving!
1) Drive around surface of Mars – take samples of items (Mars rocks):
• experimental data type = mass spectroscopy = data histogram = feature vector
2) Perform Intelligent Data Operations (Data Mining in Action):
• Supervised Learning
– Search for items with known characteristics, and assign items to known classes
• Unsupervised Learning
– Discover what types of items are present, without preconceived biases; find the set
of unique classes of items; discover unusual associations; discover trends and new
directions (new leads) to follow
• Semi-supervised Learning
– Find the rare, one-of-kind, most interesting items, behaviors, contexts, or events
3) Enact On-board Intelligent Data Understanding & Decision Support = Science Goal (or Business Goal) Monitoring:
– “stay here and do more” ; or else “follow trend to a more interesting location”
– “send discoveries to human analysts immediately” ; or “send informational results later” 26
OUTLINE • First came Data, then Big Data, now Hyper-Data
• The Self-Driving Organization
• Steps to Discovery: Data Science & Analytics
• Case Study: Mars Rovers (metaphor)
• Analytics Roadmap for Self-Driving Organizations
27
Smart Analytics for the IoT — Data-to-Action Triage • General example of IoT Analytics Triage in DDDAS: IoT Event Mining in Big Data Collections for Actionable Intelligence:
Behavior modeling (anomaly & trend detection) and ad hoc inquiry for Discovery
Identifying, characterizing, & understanding events for data-driven Decisions
Deciding which events need immediate investigation and/or intervention = Action!
… for people, products, and processes!
• Many other examples: Predictive Maintenance alerts (from ubiquitous machine / engine sensors)
Prescriptive efficiency modeling and implementation (from process mining)
Supply chain monitoring (from manufacturing & shipping sensors)
Predictive personnel & asset logistics (from RFID tracking data & trajectory modeling)
Cybersecurity alerts (from network logs)
Preventive Fraud alerts (from financial applications)
Health alerts (from electronic health records and national health systems)
Tsunami alerts (from geo sensors everywhere)
Social event alerts or early warnings (from social media) 28
The Analytics Roadmap : What We Need • Design Patterns for Streaming Data Analytics:
– Detecting POI (Person of Interest, or any object)
– Detecting BOI (Behavior of Interest from any “dynamic actor”)
– Precomputed scenarios and their responses (to speed up “best action”)
– Design Thinking : UX, CX, EX (User / Customer / Employee eXperience)
• Edge Analytics (moving the algorithms to the sensor) – Locality in Time
• Near-field Analytics (who, what is local to my asset?) – Locality in Geospace
• Related-entity Analytics (what else is like this?) – Locality in Feature Space
• Agile Analytics: DataOps, Fail-fast, Iterative, Learning Systems 29
The Self-Driving Organization: Delivering Big Value from Smart Data
It is not true that the one with the most toys (the most data) wins. The one who wins is the one who innovates the most with smart data!
Defining Characteristics Data Analytics Maturity Organizational Posture
Sensing, Streaming Descriptive Passive
Responsive Diagnostic Reactive
Learning, Agile Predictive Proactive
Contextual, Optimizing Prescriptive Predictive Reactive
Deciding, Acting Cognitive Best action, Smart
http://www.boozallen.com/datascience 30
Recommended