can -elephant bigdata dance iot internet things tune ...kirkborne.net/London2017/KirkBorne-LondonBigDataWorld2017.pdfAbstract I will describe the concept of a self-driving organization

Kirk Borne @KirkDBorne

The Self-Driving Organization and Edge Analytics in a Smart IoT World

http://caanz.co.nz/industry-voice/whos-afraid-of-big-data/ https://www.linkedin.com/pulse/can-elephant-bigdata-dance-iot-internet-things-tune-lambba

Abstract

I will describe the concept of a self-driving

organization that learns, gains actionable insights,

discovers next-best move, innovates, and creates

value from streaming big data and edge analytics

via ubiquitous data sources in the IoT-enriched

world. I will also present an Analytics Roadmap

for the Smart IoT-enabled Cognitive Organization.

2

OUTLINE • First came Data, then Big Data, now Smart Data

• The Self-Driving Organization

• Steps to Discovery: Data Science & Analytics

• Case Study: Mars Rovers (metaphor)

• Analytics Roadmap for Self-Driving Orgs

3






4

Ever since we first explored our world…

http://www.livescience.com/27663-seven-seas.html 5

…We have asked questions about everything around us.

https://jefflynchdev.wordpress.com/tag/adobe-photoshop-lightroom-3/page/5/ 6

So, we have collected evidence (data) to answer our questions,

which leads to more questions, which leads to more data collection,

which leads to more questions, which leads to BIG DATA!

y ~ 2 * x (linear growth)

y ~ 2 ^ x (exponential)

7

https://www.linkedin.com/pulse/exponential-growth-isnt-cool-combinatorial-tor-bair

Big Wow!

y ~ x! ≈ x ^ x → Combinatorial Growth! (all possible interconnections, linkages, and interactions)

https://www.linkedin.com/pulse/exponential-growth-isnt-cool-combinatorial-tor-bair

Semantic, Meaning-filled Data:

• Ontologies (formal)

• Folksonomies (informal)

• Tagging / Annotation – Automated (Machine Learning)

– Crowdsourced

– “Breadcrumbs” (user trails)

Broad, Enriched Data:

• Linked Data (RDF)

– All of those combinations!

• Graph Databases

• Machine Learning

• Cognitive Analytics

• Context

• The 360o view

What makes your data Smart?

The Human Connectome Project: mapping and linking the major pathways in the brain. http://www.humanconnectomeproject.org/

8

Internet of Things (IoT): Deploying intelligence at the point of data collection!

(Machine Learning at the edge of the network = Edge Analytics!)

Internet of Everything

https://www.nsf.gov/news/news_images.jsp?cntn_id=122028

The Internet of Things (IoT) will be an interconnected universe of Dynamic Data-Driven Application Systems (dddas.org) =>

Combinatorial Explosive Growth of Smart Data!

9

https://www.linkedin.com/pulse/can-elephant-bigdata-dance-iot-internet-things-tune-lambba

https://www.nsf.gov/news/news_images.jsp?cntn_id=122028

http://dddas.org/

• Smart Health

• Precision Medicine

• Precision Farming

• Personalized Financial Services

• Smart Organizations

• Predictive Maintenance

• Prescriptive Maintenance

• Smart Grid

• Smart Apps

• Predictive Retail

• Precision Marketing

• Smart Highways

• Precision Traffic

• Smart Cities

• Predictive Law Enforcement

• Personalized Learning

Smart => Predictive, Precision, Personalized!

Ubiquitous Smart Data in the IoT: Applications and Use Cases are everywhere…

… that demands dynamic action = Edge Analytics

10






11

What is Self-Driving? … ~Autonomous! (e.g., cars, cities, deep space probes, organizations)

Defining Characteristics Data Analytics Maturity Organizational Posture

Sensing, Streaming

Responsive

Learning, Agile

Contextual, Optimizing

Deciding, Acting

Note: additional KEY CHARACTERISTICS = Secure, Compliant, “Embedded Reporting” = explosive growth of “NC2” data (“in situ” data) that delivers CONTEXT

12


Sensing, Streaming Descriptive

Responsive Diagnostic

Learning, Agile Predictive

Contextual, Optimizing Prescriptive

Deciding, Acting Cognitive


13



Sensing, Streaming Descriptive Passive

Responsive Diagnostic Reactive

Learning, Agile Predictive Proactive

Contextual, Optimizing Prescriptive Predictive Reactive

Deciding, Acting Cognitive Best action, Smart


14


OUTLINE • First came Data, then Big Data, now Hyper-Data





15

Some Quick Definitions • Statistics = the practice (and science) of collecting and analyzing

numerical data

• Machine Learning (ML) = mathematical algorithms that learn from experience (historical data)

• Data Mining = application of ML algorithms to data

• Artificial Intelligence = application of ML algorithms to robotics and machines

• Data Science = application of scientific method to discovery from data (including statistics, machine learning, and more: visual analytics, machine vision, computational modeling, semantics, graphs, network analysis, data indexing schemes,…).

• Analytics = the products of machine learning & data science.

• Statistics = the practice (and science) of collecting and analyzing numerical data.

• Machine Learning (ML) = mathematical algorithms that learn from experience (historical data).

• Data Mining = application of ML algorithms to data.

• Artificial Intelligence (AI) = application of ML algorithms to robotics and machines.

Source for graphic: http://blogs.sas.com/content/subconsciousmusings/2014/08/22/looking-backwards-looking-forwards-sas-data-mining-and-machine-learning/

16

1) Class Discovery: Finding new classes of objects (population segments), events, and behaviors. This includes: learning the rules that constrain the class boundaries.

2) Correlation (Predictive and Prescriptive Power) Discovery: Finding patterns and dependencies, which reveal new governing principles or behavioral patterns (the “customer DNA”).

3) Novelty (Surprise!) Discovery:

Finding new, rare, one-in-a-[million / billion / trillion] objects and events.

4) Association (or Link) Discovery: Finding unusual (improbable) co-occurring associations.

Machine Learning : Algorithms that learn from experience

17

(Graphic by S. G. Djorgovski, Caltech)

5 Levels of Analytics Maturity

1) Descriptive Analytics

– Hindsight (What happened?)

2) Diagnostic Analytics

– Oversight (real-time / What is

happening? Why did it happen?)

3) Predictive Analytics

– Foresight (What will happen?)

4) Prescriptive Analytics

– Insight (How can we optimize what happens?)

5) Cognitive Analytics – Right Sight (the 360 view , what is the right

question to ask for this set of data in this

context = Game of Jeopardy)

– Finds the right insight, the right action, the

right decision,… right now!

– Moves beyond simply providing answers, to

generating new questions and new

hypotheses, for your “next-best move”

18

PREDICTIVE

Analytics

Find a function (i.e., the model) f(d,t) that

predicts the value of some predictive

variable y = f(d,t) at a future time t, given

the set of conditions found in the training

data {d}.

=> Given {d}, find y.

PRESCRIPTIVE

Analytics

Find the conditions {d’} that will produce a

prescribed (desired, optimum) value y at a

future time t, using the previously learned

conditional dependencies among the

variables in the predictive function f(d,t).

=> Given y, find {d’}.

19

PREDICTIVE

Analytics





data {d}.


PRESCRIPTIVE

Analytics







Confucius says…

“Study your past to know

your future”

20

PREDICTIVE

Analytics





data {d}.


PRESCRIPTIVE

Analytics







Confucius says…

“Study your past to know

your future”

Baseball philosopher Yogi Berra says…

“The future ain’t what it

used to be.”

21

Context is King! “It’s not what you look at that matters – it’s what you see.” – Henry David Thoreau

“You can see a lot just by looking.” – Yogi Berra

• Context is “other data” about your data = i.e., Metadata!

• The 3 most important things in your data are: Metadata, Metadata,

Metadata!

• Metadata are…

– Other Data that describes Other Data

– Other Data that describes Your Data

– Your Data that describes Other Data

• e.g., Connected “Smart” Cars = that car that is braking 3 vehicles

ahead of you = informs your vehicle to brake now!

• The Smart Business = predictive maintenance algorithm alerts the

corresponding asset, the right skilled technician, & the right tool to

converge at right place at the right time! 22

Levels of Maturity in the Self-Driving Organization








23






24

Case

Study:

Mars

Rovers (metaphor)

25

Mars Rover: intelligent data-gatherer, mobile data mining

agent, autonomous decision-support system, and self-driving!

1) Drive around surface of Mars – take samples of items (Mars rocks):

• experimental data type = mass spectroscopy = data histogram = feature vector

2) Perform Intelligent Data Operations (Data Mining in Action):

• Supervised Learning

– Search for items with known characteristics, and assign items to known classes

• Unsupervised Learning

– Discover what types of items are present, without preconceived biases; find the set

of unique classes of items; discover unusual associations; discover trends and new

directions (new leads) to follow

• Semi-supervised Learning

– Find the rare, one-of-kind, most interesting items, behaviors, contexts, or events

3) Enact On-board Intelligent Data Understanding & Decision Support = Science Goal (or Business Goal) Monitoring:

– “stay here and do more” ; or else “follow trend to a more interesting location”

– “send discoveries to human analysts immediately” ; or “send informational results later” 26





• Analytics Roadmap for Self-Driving Organizations

27

Smart Analytics for the IoT — Data-to-Action Triage • General example of IoT Analytics Triage in DDDAS: IoT Event Mining in Big Data Collections for Actionable Intelligence:

Behavior modeling (anomaly & trend detection) and ad hoc inquiry for Discovery

Identifying, characterizing, & understanding events for data-driven Decisions

Deciding which events need immediate investigation and/or intervention = Action!

… for people, products, and processes!

• Many other examples: Predictive Maintenance alerts (from ubiquitous machine / engine sensors)

Prescriptive efficiency modeling and implementation (from process mining)

Supply chain monitoring (from manufacturing & shipping sensors)

Predictive personnel & asset logistics (from RFID tracking data & trajectory modeling)

Cybersecurity alerts (from network logs)

Preventive Fraud alerts (from financial applications)

Health alerts (from electronic health records and national health systems)

Tsunami alerts (from geo sensors everywhere)

Social event alerts or early warnings (from social media) 28

The Analytics Roadmap : What We Need • Design Patterns for Streaming Data Analytics:

– Detecting POI (Person of Interest, or any object)

– Detecting BOI (Behavior of Interest from any “dynamic actor”)

– Precomputed scenarios and their responses (to speed up “best action”)

– Design Thinking : UX, CX, EX (User / Customer / Employee eXperience)

• Edge Analytics (moving the algorithms to the sensor) – Locality in Time

• Near-field Analytics (who, what is local to my asset?) – Locality in Geospace

• Related-entity Analytics (what else is like this?) – Locality in Feature Space

• Agile Analytics: DataOps, Fail-fast, Iterative, Learning Systems 29

The Self-Driving Organization: Delivering Big Value from Smart Data

It is not true that the one with the most toys (the most data) wins. The one who wins is the one who innovates the most with smart data!







http://www.boozallen.com/datascience 30

Documents

can -elephant bigdata dance iot internet things tune ...kirkborne.net/London2017/KirkBorne-LondonBigDataWorld2017.pdfAbstract I will describe the concept of a self-driving organization