Embed Size (px)
Citation preview
Databases and graph analysis
back to the future?
Krystian Piećko CTO & Co-founder PiLab S.A. 2015-09-30
Introduction2
Few words about PiLab
Mother of God…
3
Databases in Enterprise Environment
Theory vs. PractiseHow enterprises are using data
5
Two data modellers in the same roomBetter fight than Pride with Nastula
6
Never forget about it7
NoSQL
NoSQL rises 2009
• SQL is hard • RDBMS can’t fetch data
so fast • Schemas are for
grandpas • Who needs SQL - we have
got Java and Python • Analytics is to slow on
RDBMS • Unstructured is growing • other?
9
It is very simple to dim the lights in server room
10
Power to the userLimited access is the problem
11
NoSQL in practice12
NoSQLThere is less java/python/whatever programmers than
people that know how to use SQL13
NoSQL in practiceDo you all know this?
14
Schema vs. schemaless15
NoSQL with SQL interfaceMany of the new ones are missing here
16
After few yearsSome people tried to change the meaning to Not Only
Short learning cycle :)17
NewSQL
And it has begunThe shift showed the “old school” db guys, that there is a
chance on the marketplace19
Inefficient query that runs agains NewSQL is still inefficient but it runs faster :)
20
Truth ;)21
The largest RDBMS implementation I know
22
Facebook facts 2014
• 60k+ servers • MySQL database
cluster • Memcache • 2.4+ billion pieces of
content and 750TB+ of data every day
• 35TB+/h daily ingest to HPVertica with SLA on that
• 300+ identical nodes with 10Gigs/s connectivity
23
Probably trueThe cost vs. value
24
Only some industries need itall need to think about it
25
Database size of 90% of major vendor DB usage is less than 900GB
26
Graph analysis
Graph is how we think28
6 degrees of separation29
6 degrees of separation30
Cluster/filter/optimize31
Graphs are growingEntity relationship model is the graph
32
Investigations
Techniques• There are no official data
about all vendors and the software just rumours
• Software that is being used was i2, Palantir, other custom made
• Huge amount of work on unstructured data
• 7 ways to write Al-Qaeda • How to investigate tip? • Manually tagging
unstructured data • Social media cross join
34
Being able to connect the dotsSharing the investigation information was
35
Field investigation was the clue
36
Clue to the case was the visualisationand ad-hoc querying by not technical users
#SELECT * FROM users WHERE clue > 0
37
Banks
~40th Bank in UShas the size of the largest Polish Bank
39
IT companies in SV40
Almost every part of the banking industry market has been taken
Only few spots left
41
Minimum 10 different softwares involvedRegulatory dominant issues
Alerting systemInvestigation systemCase management
system
Twist & Tune
//
//
42
About PiLab
Technology44
Technology is not a product
45
Product is something that solves the customer specified need and customer
wants to pay for it46
Finding a niche47
Short learning cycle48
User with the data49
Demo?50
Wykorzystane strony • http://database.ca/databasehumorhumourjokes.aspx • https://smist08.files.wordpress.com/2012/01/nosql.png • https://plus.google.com/+YaiEakwattana/posts/Lda6xNR7LV9 • http://image.slidesharecdn.com/sqlonhadoop-150617075749-lva1-app6892/95/sql-on-hadoop-6-638.jpg?
cb=1435155882 • http://vertica.tips/2014/08/27/how-vertica-met-facebooks-35tb-per-hour-ingest-sla/ • http://www.quickmeme.com/p/3vxe25 • http://image.slidesharecdn.com/thefivegraphsoffinance-graphconnectnyc-131108130053-phpapp02/95/the-
five-graphs-of-finance-philip-rathle-and-emil-eifrem-graphconnect-ny-2013-30-638.jpg?cb=1383915667 • http://www.koreatimes.co.kr/upload/news/100108_p06_cartoon.jpg • http://cnt13.blogspot.com/2013/02/terrorist-networks-look-into-911-tragedy.html • http://belfastgroup.digitalscholarship.emory.edu/static/img/2013-dh-graph.png • http://www.datamodel.com/index.php/tag/database-design/ • http://www.koreatimes.co.kr/upload/news/100108_p06_cartoon.jpg • http://www.frugaldaddy.co.uk/images/splash-guard.jpg • http://img06.deviantart.net/bf9b/i/2013/301/4/1/funny_chuck_norris_converse_shoe__by_rickfrost-d6s4kqz.jpg • http://www.weevermedia.com/wp-content/uploads/2015/08/fraud-mobile-app-marketing-300x241.jpg • http://www.b-eye-network.com/blogs/rogers/thanksgiving.jpg • http://www.neatorama.com
52
