View
4
Download
0
Category
Preview:
Citation preview
Towards Sustainable Insights
Tim Kraska <tim_kraska@brown.edu>
http://www.huffingtonpost.co.uk/2016/01/08/a-glass-of-red-wine-is-the-equivalent-to-an-hour-at-the-gym-says-new-study_n_7317240.html
ANewStudyshows:AGlassOfRedWineIsTheEquivalentToAnHourAtTheGym[FoxNews02/15andothers]
Anewstudyshows:Secrettowinninganobel prize?EatMoreChocolate[Time 10/12]
Anewstudyshows:Secrettowinninganobelprize?EatMoreChocolate[Time 10/12]
Scientistsfindthesecretoflongerlifeformen(Thebadnews:castrationisthekey) [DailyMailUK,09/12]
http://www.dailymail.co.uk/sciencetech/article-2207981/Scientists-secret-living-life-men-bad-news-Castration-key.html
There has been an explosion of (data-driven) discoveries, many of which being questionable.
Reasons are manifold, but…thedatabasecommunity
… andmanyothers
workshardontobenotleftout(again)
AnoteforReviewer2:Weactuallylikedyourcommentsandithelpedustosharpenourpoints.Ifyoufeelinanywayoffendedbythistalk,thiswasnotmyintentionandIammorethanhappytomakeituptoyouwithalotofwhisky.Justcometomeafterthetalkandsayweneedtodrink.Knowingthiscrowd,enoughpeoplewilldoitandIwillevenneverfindoutyouridentityifyoudonotwishso.
Let me introduce (virtual) Reviewer 2:
Thepaper'sshortcomingsareinitsmotivation,solution,andpresentation.
ThepartofthepaperthatIdidlikewastheexamplesgiveninSec
2.2.2.
OutlinePart I: The problem with:
A. Interactive Data Exploration
B. Visualization Recommendation Systems
C. Hypothesis Generator
A. Part II: Solutions
A) Interactive Data Exploration Tools (Vizdom as an Example)
Why Visualizations contribute to the problemIf a visualization provides any insight, it is an hypothesis test (just one where you not necessarily know if it is statistical significant)
Otherwise, visualizations have just to be taken as pretty pictures about (potentially) random facts
gender
coun
t
Male Female Other
A
salary over 50k
coun
t
True Falsegender
coun
t
Male Female Other
gender
coun
tMale Other
salary over 50k
coun
t
True False
gender
coun
t
Male Female Other
B C
education
coun
t
HS Bachelor Master PhD
marital status
coun
t
Married NeverMarried
NotMarried
Widowed
Female
salary over 50k
coun
t
True False
education
coun
t
HS Bachelor Master PhD
marital status
coun
t
Married NeverMarried
NotMarried
Widowed
age
coun
t
10 20 30 50 60 7040 9080
ageco
unt
10 20 30 50 60 7040 9080
0.011p
t-test
D
E F
salary over 50k
coun
t
True False
education
coun
t
HS Bachelor Master PhD
marital status
coun
t
Married NeverMarried
NotMarried
Widowed
gender
coun
t
Male Female Other
A
salary over 50k
coun
t
True Falsegender
coun
t
Male Female Other
gender
coun
t
Male Other
salary over 50k
coun
t
True False
gender
coun
t
Male Female Other
B C
education
coun
t
HS Bachelor Master PhD
marital status
coun
t
Married NeverMarried
NotMarried
Widowed
Female
salary over 50k
coun
t
True False
education
coun
t
HS Bachelor Master PhD
marital status
coun
t
Married NeverMarried
NotMarried
Widowed
age
coun
t
10 20 30 50 60 7040 9080
age
coun
t
10 20 30 50 60 7040 9080
0.011p
t-test
D
E F
salary over 50k
coun
t
True False
education
coun
t
HS Bachelor Master PhD
marital status
coun
t
Married NeverMarried
NotMarried
Widowed
If visualizations are used to find something interesting, the user is doing multiple hypothesis testing
Running Example: Survey on Amazon Mechanical Turk
Our goal: To find good indicators (correlations) that somebody knows who Mike Stonebraker is.
And after searching for a bit, one of my favorites
Pearsoncorrelationsignificance-levelp<0.05
But Why Does the DB community make the situation worse?
So What Did Reviewer 2 say?Blamingthemultiple-comparisonproblemonfastvisualization-generationislikeblamingfastcarsforchilddrivercasualties
duetocaraccidents…
But…
2) Visual Recommendation Systems (SeeDB as an Example)
0
0.2
0.4
0.6
0.8
1
V1 V2
Normalize
dAg
gr(CollumnA)
CollumnB(filteredColumnC=V?)
Target
0
0.2
0.4
0.6
0.8
1
V1 V2
Normalize
dAg
gr(CollumnA)
CollumnB(filteredColumnC=V?)
Reference
0
0.2
0.4
0.6
0.8
1
V1 V2
Normalize
dAg
gr(CollumnA)
CollumnB(filteredColumnD=V?)
Target
Uninteresting
Interesting
What is different
The system automatically generates thousands of visualizations and ranks them somehow (e.g., based effect size)
SeeDB on Our Survey Data
Startup CorporationFilter: All
0
0.2
0.4
0.6
0.8
% C
hedd
ar &
Sou
r Cre
am Potato Chips vs Workspace Preference
Startup CorporationFilter: Belief in Alien Existence
0
0.5
1
% C
hedd
ar &
Sou
r Cre
am Potato Chips vs Workspace Preference
Startup CorporationFilter: Disbelief in Alien Existence
0
0.2
0.4
0.6
0.8
% C
hedd
ar &
Sou
r Cre
am Potato Chips vs Workspace Preference
Startup CorporationFilter: Prefer Blow Hair Drying
0
0.1
0.2
0.3
0.4
% C
hedd
ar &
Sou
r Cre
am Potato Chips vs Workspace Preference
…Ididlike[…]theexample…
What is the Problem?
The user is in the dark what the system did. The system might have “tested” thousands of potential visualization, just to find something interesting.
What did Reviewer 2 say?
Thesesystemsarenotdesignedforanaveragepersontorunandgetinsightsthattheycanpublishmedicalarticleson!The
endusersarestillanalysts.TheonlydifferenceisthattheyautomatehypothesesgenerationandNOThypotheses
testing,…
WARNINGAfter using the tool,throwaway thedata.
It is not safe!1
My suggestions, papers should include in the future a a warning like
1To be more precise: you do not have to throw it all away, but you can not use the same data anymore for significance testing
3) Real Hypothesis Generators(Data Polygamy as an Example)
(Data) Polygamy is bad, especially if you do not know what is going on.
OutlinePart I: The problem with:
A. Interactive Data Exploration
B. Visualization Recommendation Systems
C. Hypothesis Generator
A. Part II: Solutions
Should we stop working on IDE, Recommenders, etc?
• Actively inform the user about the risk factors
• Try your techniques over random data with different data sizes
• If possible, split data into a exploration and a validation set. • Be aware, significantly lowers the power • Everything on the validation data set has to be carefully handled (i.e., use
multi-hypothesis control)
• If possible, use additional experiments (e.g., A/B testing)• Requires a small number of hypothesis and careful design• Might not always be possible or is very expensive
Better: control the multi-hypothesis problem from the start
NO
QUD
EQuantifyingtheUn
certaintyin
DataExploratio
n
Python
BigDAWG
IDEAInteractiveDataExploration
Accelerator
LegacySystems
Mlbase2
With
hypothesis
control
OurInteractiveDataExplorationStack(BIDES)
Many Interesting Open Problems
• Transparent hypothesis testing how to automatically derive what the hypothesis is the user is testing
• How to convey the meaning to the user(e.g., FDR vs family-wise error)
• Safe recommender techniques(we are currently exploring new techniques based VC-dimensions to control the error)
• Incremental multiple-hypothesis control techniques (for example, see ”Controlling False Discoveries During Interactive Data Exploration” CoRR abs/1612.01040 how we use new alpha-investing policies to do that)
• Dependencies between hypothesis (this can safe ”hypothesis budget”)
• …
Wearejustatthebeginning
A Final Note from Reviewer 2 onIs the Situation really so Bad?
..,thesystemsthatarecriticizedbythispaperareessentiallythree
tools[4,6,28]… Sotheproblemisnotreallyasseriousasitmightseemasnoneofthesesystemsareusedbyanyoneinpractice
Tim Kraska <tim_kraska@brown.edu>
Specialthanksto:
AlastnotetoReviewer2:1st Isincerelyhopeyouarenotoneofmyletterwritersformytenurecase:)2nd Yourcommentsactuallyhelpedustoimprovethepaperandhelpedwiththetalk.Sothankyou!3rd Iamhappytopayforyourdrinkstonighttomakeituptoyou.
Recommended