Machine Learning as The Key Ingredient to
“Self-Driving” Data Center Sergey A. Razin
CTO, SIOS Technology Corp.
2
Who is SIOS?
Who is SIOS?
3
How did IT
start?
How did IT start?
5
How did IT started?
6
How did IT started?
7
9
10
API
11
Operational Scale & Complexity in Virtualized IT Underlies the Problem
12
# of Data Points for 7 day (1 week) (over 60B for 10,000 VMs)
VMs
# of Objects
# of Metrics
0
10,000,000,000
20,000,000,000
30,000,000,000
40,000,000,000
50,000,000,000
60,000,000,000
70,000,000,000
1001,000
3,0005,000
10,000
Axi
s Ti
tle
100 1,000 3,000 5,000 10,000
VMs 100 1,000 3,000 5,000 10,000
# of Objects 2,000 15,000 50,000 100,000 200,000
# of Metrics 604,800,000 4,536,000,000 15,120,000,000 30,240,000,000 60,480,000,000
13
What is Machine
Learning?
Does ML exist today?
14
Gap In Tools
15
16
17
Goodbye, Big Data!
Hello, Smart Data!
We have a problem here…
We can’t see it UNLESS we have tools
Conventional Tools
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 19
Data
Still Dealing With Noise
Conventional Tools
State of The Productized Art
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 20
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 21
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 22
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 23
What is the problem? Why Conventional Tools Can’s Solve It?
• Understanding of Data Semantics because: • Data is Big
• Data is complex
• Data is high-dimensional (with a lot of interrelationships and interplay)
• Data is features rich
• individual measurable property of a phenomenon being observed (examples of the data features: latencies, IOPS, etc.)
• Solution required different type of tool
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 24
What if there was a solution?
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 25
Topological Behavior Analysis
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 26
Topological Data Analysis (TDA)
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 27
TDA
Mathematics (Topology) Data
Shape
Topological Data Analysis (TDA)
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 28
NOISE
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 29
Topological Data Analysis (TDA)
DATA HAS SHAPE
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 30
DATA HAS MEANING
Topological Data Analysis (TDA)
Now back to user’s problem…
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 31
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 32
Topological Behavior Analysis (TBA)
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 33
Topological Behavior Analysis (TBA)
IT Operations Is A Big Data Problem
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 34
Noise
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 35
SIOS iQ – Applying Big Data Principles Topological Behavior Analysis
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 36
Topological Behavior Analysis & Machine Learning
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 37
Topological Behavior Analysis & Machine Learning
Copyright @ 2015 SIOS Technology Corp. All rights reserved. 38
Topological Behavior Analysis (TBA)
39
What’s Next?
Self-Driving Datacenter
40
41
Why should IT
brush up on
Algorithms?
Expert Knowledge + Machine Intelligence
42
43
Stay Relevant - Education
Machine Learning Course: https://www.coursera.org/learn/machine-learning
44
Stay Relevant - Education
Data Scientist Certification Program: https://www.coursera.org/specialization/jhudatascience/1
DIY – Machine Learning
http://www.r-project.org/
http://www.cs.waikato.ac.nz/ml/weka/
http://madlib.net/
https://mahout.apache.org/
http://jubat.us/en/
46
How about those latencies?
Anomaly Detection – K-means Clustering
47
Outlier Detection Using K-means
Anything outside 3 standard deviations is an outlier
Social Graph -> IT Graph
http://neo4j.com/
FlockDB
https://github.com/twitter/flockdb
http://giraph.apache.org/
49
Neo4j
https://www.vmware.com/support/pubs/sdk_pubs.html
API
50
Want to see it
in Action? http://us.sios.com/
SIOS iQ™ Machine Learning Analytics for optimizing
VMware environments
52
SIOS iQ™ Machine Learning Analytics for optimizing VMware environments
Participate in our SIOS iQ™
usability study and have a coffee on us!
53
• Always Interested
Job Opportunities @ SIOS
Great People. Exciting and Innovative Projects.