Snapshot All Too Gooood

8/14/2019 Snapshot All Too Gooood

1/37

SNAPDIFF Comprehension for coherent measurements1

DB2 APD : Enabling Data Analysis

Dominique Evans, DB2 Avance Pro!"em Determination, IB# evans$ca%i!m%com
mailto:[email protected]:[email protected]


2/37

2

2

&afs&tor&u&evans&pu!"ic&!in&snapiff

Presentation goals

PA'( I Ientif) the pro!"em

Demonstrate simp"e SNAPDIFF

Introuce notion of section *+roup

Fi"ters

PA'( II Comp"e- SNAPDIFF

Importe fi"es

.oine fi"es

#eta ata an formu"ae

/oa an un"oa

/afs/tor/u/evansd/public/bin/snapdiff


3/37

0

0


The problem

1% E-periments create "ar+e amounts of measurement ata

2% Data is hetero+eneous%

0% sefu" ata *a"most a"a)s invo"ves mu"tip"e measurementcaptures over time interva"s% *e%+% snapshots%

3% (he num!er of atasets rapi") e-cees human a!i"it) to +ainmuch insi+ht or 4no"e+e from the ra fi"es

5% Automate ata capture scripts can easi") +enerate o6ens orhunres of reports


4/37

3

3


Snapshots

Snapshots

!2p

!2p

7ther

Snapshots

Snapshots

!2p

!2p

7ther

Snapshots

Snapshots

!2p

!2p

7ther

(ime 8 (ime 91 (ime 92

:


5/37

5

5


A Key to analysis

;hat is chan+in+ over time Q , to create name > va"ue pairs %


13/3710

10



14/3713

13


What are the delta &alues'


15/3715

15


A richer e%ample. Analysis o" slo! (AD) secondary.

snapiff U%poo"K8LK8LVoin poo"si6esimport U%!K8LK8Limport U%eusnapshot8K8LK8Lmeta &(#P&evans&meta

un"oa poo"!eu

snapiff roh T/o+ +ap runnin+ avera+eT48"oa poo"!eu

on")pivotpf Tper seconprefetch%UsecDe"ta Num!er rite "o+ I7sPrimar) Stan!) E-tent /a+Interva" compute "o+ +apW


16/371=

1=



17/371G

1G


A di""erent &ie!, same data:

snapiff corr T/o+ +ap runnin+ avera+eT48 "oa poo"!euon")pivotpf TDe"ta%UC7DS1%U!2shre%Utota"De"ta%UC7DS1%U!2reo%Utota"De"ta%UC7DS1%U!2hars%Utota"T on")+"o!a"


18/371H

1H



19/371

1


Bac$ to snapdi"" basics.

Simp"est usa+e is 2 fi"es, e%+% 2 ata!ase snapshots

Defau"t isp"a) is !) atapoint ro,

#easurements in co"umnsi%e% time a"on+ - a-is *snapiff etects snapshot timestamps


20/3728

28


*impli"y. +et rid o" sections. *ort . +et rid uninteresting data.


21/3721

21


E%actly the same, but delta display -includes percent by de"ault


22/37

22

22


What is /section0 ' -a simple /group by0

?arious snapshots have repeatin+ +roups

Snapiff automatica"") etects certain +roups *ata!ase, !ufferpoo",ta!"espace%

Datapoints ithin sections are qua"ifie !) the section name

Can turn off ith nosection

Can ec"are an) atapoint a section e"imiter usin+

section Qmeasurement nameW

*here Qmeasurement nameW is a re+e- that i""" !e use to ientif) aatapoint name so ma4e it unique


23/37

20

20


Ta$e a bu""erpool snapshot "or instance:

1se a restricti&e "ilter -"or clarity.

section /Bu""erpool name0 is automatically detected.


24/37

23

23


*o !hat is p"' -abbre&iation o" p"ilter

/Post -data ingestion "ilter0.

A fi"ter app"ie after a"" atasets are in+este *an have ha metaru"es app"ie if present%

A"so f *fi"ter , operates on incomin+ ataset *roppin+ atapointsthat o not qua"if)% Ear"iest fi"ter%

f an pf usua"") ientica" resu"ts un"ess usin+ fancier options *meta, import %

f is faster *since rops the ata much sooner , !ut "oss) for metaprocessin+%

sfi"ter Xre+e-pM , fi"ters ho"e sections%

e%+%J


25/37

25

25


3iltering sections

B(;J formatte ata can !e turne offJ noformat


26/37

2=

2=


import . (o! to e%tend the data.

As QneW ata to primar) atasets

e%+% import ata!asesnapshot to !ufferpoo"snapshot

Sections not reco+ni6e *mi+ht&pro!a!") i"" enhance this

A"" importe ata is Q+"o!a"W *unqua"ifie re+ar"ess of interna" repeatin++roups%

Dup"icates in import atapoints roppe

An) name > va"ue ata can !e importe

Data shou" !e coherent *i%e% re"ate

An) num!er of separate sources can !e importe at once

Can import one ataset *e%+% constants, or count* primar) atasets*coherent measuremets


27/37

2G

2G



28/37

2H

2H


4oin is an import !ith relationship through the section

A""o ata to !e ae to section specific ata, *rather than

+"o!a"") as ith import

oo e-amp"e is !ufferpoo" snapshot@ since the ra snapshot

oes not inc"ue pa+e si6e * %

A QBufferpoo" Pa+e si6eW atapoints usin+ Voin%

A""os correct computation of !)tes from ra snapshot ata%


29/37

2

2


4oin : +oing bac$ to the bu""erpool snapshots e%ample

CaveatJ Do not use nosame


30/37

08

08


Primary data duplicate processing

Choice of severa" po"iciesJ

up K i+nore ha"t rep"ace a++re+ate se+re+ate Y *efau"t se+re+ate

1% I+nore *rops ups

2% Za"t *stops ith error on up

0% 'ep"ace *so "ast up ins

3% A++re+ate *sum

5% Se+re+ate *automatica"") +enerate moifie name *DPn


31/37

01

01



32/37

02

02


P567T &ie!s

reat for o!servin+ "ar+e num!ers of measurements

Best ith ver) se"ective fi"ters *so to fit on the screen


33/37

00

00


8eta measurements , and "ormulae

A""o e-pressions to !e app"ie to measurement ata

7perans are atapoint names

Eva"uate on a per ataset !asis

AF(E' f *fi"ter % BEF7'E pf *pfi"ter

Each ataset can reference any pre&iousataset

measurement*s


34/37

03

03



35/37

05

05



36/37

0=

0=



37/37

0G

9A*T *95DE

*T7P (E)E

snapiff < for comp"ete *more or "ess he"p

!2service ;i4i comin+ soon%

/ocation of script *Per"J


Documents

Snapshot All Too Gooood