Data ScienceSoftware Platform

The Solitude Of The Data Team Manager

Data Driven NYC 04/11/16

My nerdy background

Type Systems Automated Proving Abstract Program Interpretation Functional Programming Garbage Collection and Vms

Graph Analytics Chess IA Natural Language Processing 80% Emacs / 20% VIM

So to sum it up …

3

I (USED TO) HATEGUIs

…and our software is

A supercharged Visual IDE for Data Teams

Deployment

Meet HAL

Hal AlowneBI ManagerDim’s Private Showroom

Meet Hal’s Boss, DIM

Hey Hal ! We need a big data platform like the big guys.Just do what they’re doing!

‟

”Big DataCopy Cat Project

TECHNOLOGYDisconnect

What technologies should I use ? ‟ ”

Welcome to Technoslavia !

8

TOY PLATFORM ANTI-PATTERN

9

Test and Invest in Infrastructure == Skilled Peopleor

Go For Cloud / Packaged Infrastructure

YourBrandNewHadoopClusterisperceivedasslow,notsousedandnotreliable

TECHNO MISMATCH ANTI-PATTERN

10

Assume Being Polyglotor

Be a Dictator

VS

VS

ThePythonClan

TheRTribe

TheOldElephantFraternity

TheNewElephantClub

PREDICTIVE ANALYTICS DEPLOYMENT STRATEGY

11

Website2000’winners

Companiesthatwereabletorelease fast

"ArtificialIntelligencewithDataforInternetofThings"2010’winners

Companiesabletoputintelligenceinproduction

?

Design a way to put “PREDITICTIVE MODELS” IN PRODUCTION

PEOPLEDisconnect

Who should I hire ?‟ ”

Classic Business Intelligence Team Organization

Business Leader Data Consumer

Line-of-business Data Consumer Business Project

Sponsor

BI Solution Architect

Model Designer

ETL Developer

Dashboard / Report Designer

SpecsDim

Big Boss

Data Science Team Organization

Business Leader Data Consumer

Line-of-business Data Consumer

Business ProjectSponsor

Data Engineer

Data Analyst

System Engineer / Data Architect

Business Needs

Data Scientist

ITConstraints

I.T.

Manage Expectations

15

Data Plumberer

DataEngineer

Data Scientist

Data Waiter

DataCleaner

DataAnalyst

REALJOB

DREAMJOB

Managing Extreme Personalities

16

Data Scientist

Highly Creative

Passionate

Hard to hire?

Hard to manage?

Want to take Hal’s job? Ambitious

Hard to retain?

Paired for Data

17

Data AnalystDiscover Patterns

Data EngineerMake things work

Fightdata entropy

Fighttech

entropy

What do you prefer?

18

One AnalystOne EngineerOne Data Scientist

Four data scientists

OR

Two Mindsets Can Coexist

CLICKERS CODERS

DATADisconnect

What about data ?‟ ”

What is the main reason for data project to fail ?

21

> DATA NOT AVAILABLE

BUT FOR ONLY INCREMENTAL GAIN

50 30 20

0% 25% 50% 75% 100%

Contribution totheoverallprojectperformance

BusinessGoalDefinitionandData FeatureEngineering Algorithm

How to Get Data if you don’t have it

23

THECICADA THESPIDER THEFOX

Be Optimistic !

Wait for Open Data InitiativesAnd data available in the Enterprise Hub / Data Lake !

Create Network !

Create a set of trackers or Addictive Data Collection internallyTo get Data on your side !

Hunt for Big Problem!

Convince the CEO that you can Solve a Business Critical problem And use it as an excuse to get allThe data you want !

PRODUCTDisconnect

What is Big Data about ?

25

The Age Of Distributed Intelligence

26

Global,PersonalisedandRealTimeDataDrivenServices

Data to Visualize or Data to Automate ?

2013 2014 2015 2016 2017 2018

Moving to a world of automated decision making

27

DATA FOR MORE INSIGHTS

DATAFOR AUTOMATED DECISIONS

Involve Product Team

28

ProductFeaturePersonalised ItemRanking

ProductFeatureNotifyUserOnlywhenNeeded

ProductFeature:HistoricalDataForPathOptimisation

Have Product Management Deeply Involved In the Data Team

Focus on your added value

29

Build by the Data Team

Is the problem at the Core of my Business Process?

Is it a commonproblem / with share data?

Can i solve it on my own?

Really?

Hire Consultant and Learn

Build bythe Data Team

Go for Best of BreedSaaS Solution

Build bythe Data Team?

YesNo

No Yes No Yes

No Yes

Create an API culture

Do not shareo Random Piece of Codeo Flat Fileo Email

Do shareü Reproductible documented workflowsü Clean, documented APIs

Did Hal found his solutions ?

Technology

Data

People

Product

Polyglot on top of open source

Find a way to make clickers and coders work together

Create an API culture and involve the product teams

Hunt for Big Problems and Convince the CEO

Is this the end ?‟”Hal AlowneBI ManagerDim’s Private Showroom

That was the (romanced) story…

data scientists and engineers25

2 locations

1 software

by the numbers

For a simple software for clickers and coders

Car Sharing Worldwide

Leader

Flash SalesWorldwide

Leader

One Mission : Never leave Hal ALONE

3,700 Hotels

Worldwide

2500 lovelyusers

by the numbers

70 customers

Food forthoughtswww.dataiku.com/blog

THANKYOU!

FREE (as in Beer) Software www.dataiku.com/dss