Big Data Profiling

Big Data Profiling Fribourg May 2014

Felix Naumann

The Hasso Plattner Institute

■ Founded in 1998 as a Public Private Partnership

■ Hasso Plattner, co-founder of SAP, endowed over 200 Mio. Euro.

■ Adjoined with the University of Potsdam

■ 500 students

□ BA, MA, PhD

2

■ Enterprise Platform and Integration Concepts

■ Internet Technologies and Systems

■ Human Computer Interaction

■ Computer Graphics Systems

■ Operating Systems and Middleware

■ Business Process Technology

■ Software Architecture

■ Information Systems

■ System Engineering and Modeling

■ School of Design Thinking

Felix Naumann | Data Profiling | CUSO 2014

Research Topics

■ Data Profiling and Analytics

■ Data Quality and Data Cleansing

■ Similarity Search and ETL Management

■ Knowledge Discovery and Text Extraction

■ (Linked) Open Data Integration

■ For more information on research topics and on teaching, please

see http://www.hpi.uni-potsdam.de/naumann/home.html

3


http://www.hpi.uni-potsdam.de/naumann/home.html



Profiling in Spreadsheets


4


5


6


7


8


9


10

Many interesting questions remain

■ What are possible keys and foreign keys?

□ Phone

□ firstname, lastname, street

■ Are there any functional dependencies?

□ zip -> city

□ race -> voting behavior

■ Which columns correlate?

□ county and first name

□ DoB and last name

■ What are frequent patterns in a column?

□ ddddd

□ dd aaaa St


11

Definition Data Profiling

■ Data profiling is the process of examining the data available in an

existing data source [...] and collecting statistics and information

about that data.

Wikipedia 09/2013

■ Data profiling refers to the activity of creating small but

informative summaries of a database.

Ted Johnson, Encyclopedia of Database Systems

■ A fixed set of data profiling tasks / results


12

„Big“ Data Profiling or How big is „Big“?

Data profiling = measuring the „Vs“

■ Volume

□ Row counts, etc.

■ Velocity

□ Temporal profiling

■ Variability

□ How difficult to

integrate and analyse

■ Veracity

□ How good is it?

■ …


13

Big Data

Volume

Velocity

Variety

Veracity

Viscosity

Virality

Use Cases for Profiling

■ Query optimization

□ Counts and histograms

■ Data cleansing

□ Patterns, rules, and violations

■ Data integration

□ Cross-DB inclusion dependencies

■ Scientific data management

□ Handle new datasets

■ Data inspection, analytics, and mining

□ Profiling as preparation to decide on models and questions

■ Database reverse engineering

■ Data profiling as preparation for any other data management task


14

Classification of Traditional Profiling Tasks


15

Data

pro

filing

Single column

Cardinalities

Patterns and data types

Value distributions

Multiple columns

Uniqueness

Key discovery

Conditional

Partial

Inclusion dependencies

Foreign key discovery

Conditional

Partial

Functional dependencies

Conditional

Partial

Single-column vs. multi-column

■ Single column profiling

□ Most basic form of data profiling

□ Often part of the basic statistics gathered by DBMS

□ Discovery complexity: Number of values/rows

■ Multicolumn profiling

□ Discover joint properties

□ Discover dependencies

□ Discovery complexity: Number of columns and number of

values


16

Scalable profiling

■ Scalability in number of rows

■ Scalability in number of columns

□ “Small” table with 100 columns:

2100 – 1 = 1,267,650,600,228,229,401,496,703,205,375

= 1.3 nonillion column combinations

◊ Impossible to check or even enumerate

■ Possible solutions

□ Scale up: More RAM, faster CPUs

◊ Expensive

□ Scale in: More cores

◊ More complex (threading)

□ Scale out: More machines

◊ Communication overhead

□ Intelligent enumeration and aggressive pruning


17

Challenges of (Big) Data Profiling


18

■ Computational complexity

□ Number of rows

□ Number of columns (and column combinations)

■ Large solution space

■ New data types (beyond strings and numbers)

■ New data models (beyond relational): RDF, XML, etc.

■ New requirements

□ User-oriented

□ Interactive

□ Streaming data

Agenda

19

■ Basic statistics

■ Functional dependencies

■ Keys and foreign keys

■ Data profiling tools

■ Advanced profiling


Cardinalities, distributions, and patterns

Category Task Description Cardinalities num-rows Number of rows

value length Measurements of value lengths (min, max, median, and average)

null values Number or percentage of null values distinct Number of distinct values; aka “cardinality” uniqueness Number of distinct values divided by number of rows Value distributions histogram Frequency histograms (equi-width, equi-depth, etc.)

constancy Frequency of most frequent value divided by number of rows

quartiles Three points that divide the (numeric) values into four equal groups

soundex Distribution of soundex codes

first digit Distribution of first digit in numeric values; to check Benford's law

Patterns, data types, and domains basic type Generic data type: numeric, alphabetic, date, time

data type Concrete DBMS-specific data type: varchar, timestamp, etc. decimals Maximum number of decimal places in numeric values precision Maximum number of digits in numeric values patterns Histogram of value patterns (Aa9…)

data class Semantic, generic data type: code, indicator, text, date/time, quantity, identifier, etc.

domain Classification of semantic domain: credit card, first name, city, phenotype, etc.


20

Data types and value patterns

■ String vs. number

■ String vs. number vs. date

■ Categorical vs. continuous

■ SQL data types

□ CHAR, INT, DECIMAL, TIMESTAMP, BIT, CLOB, …

■ Domains

□ VARCHAR(12) vs. VARCHAR (13)

■ XML data types

□ More fine grained

■ Regular expressions (\d{3})-(\d{3})-(\d{4})-(\d+)

■ Semantic domains

□ Adress, phone, email, first name


21

Incre

asin

g s

em

antic

s

An Aside: Benford Law Frequency (“first digit law”)

■ Statement about the distribution of first digits d in (many)

naturally occurring numbers:

□ 𝑃 𝑑 = 𝑙𝑜𝑔10 𝑑 + 1 − 𝑙𝑜𝑔10 𝑑 = 𝑙𝑜𝑔10 1 + 1𝑑

□ Holds if log(x) is uniformly distributed


22

0

20

40

1 2 3 4 5 6 7 8 9

Examples for Benford‘s Law

■ Surface areas of 335 rivers

■ Sizes of 3259 US populations

■ 104 physical constants

■ 1800 molecular weights

■ 5000 entries from a mathematical handbook

■ 308 numbers contained in an issue of Reader's Digest

■ Street addresses of the first 342 persons listed in American Men of Science


23

Heights of the 60 tallest structures

http://en.wikipedia.org/wiki/List_of_tallest_buildings_and_structures_in_the_world#Tallest_structure_by_category

Agenda

24







Naive Discovery Approach

■ Functional dependency „X → A“: whenever two records have the

same X values, they also have the same A values.

■ Given relation R, detect all minimal, non-trivial FDs X → A.

■ For each column combination X

□ For each pair of tuples (t1,t2)

◊ If t1[X\A] = t2[X\A] and t1[A] t2[A]: Break

■ Complexity

□ Exponential in number of attributes

□ times number of rows squared


25

Tane – General Idea [HKPT99]

■ Two elements of approach

1. Reduce column combinations through pruning

◊ Reasoning over FDs

2. Reduce tuple sets through partitioning

◊ Partition tuple IDs according to attribute values

◊ Level-wise increase of size of attribute set

● Consider sets of tuples whose values agree on that set


26

Discovery strategy

■ Bottom up traversal through lattice

□ only minimal dependencies

□ Pruning

□ Re-use results from previous level

■ For a set X, test all X\A → A, AX

□ only non-trivial dependencies

□ Test on efficient data structure


27

A B C D

AB AC AD BC BD CD

ABC ABD ACD BCD

ABCD

Functional Dependencies: State of the Art


28

Partial and conditional dependencies

■ Partial dependency: dependencies that do not perfectly hold

□ For all but 10 of the tuples

□ Only for 90% of the tuples

□ Only for 1% of the tuples

■ Partiality also for patterns, types, uniques, and other constraints

■ Given a partial dependencies: For which part does it hold?

□ Expressed as a condition over the attributes of the relation

■ Problems:

□ Infinite possibilities of conditions

□ Interestingness:

◊ Many distinct values: less interesting

◊ Few distinct values: surprising condition – high coverage

■ Useful for

□ Integration: cross-source condition inclusion dependency


29

Agenda

30 ■ Basic statistics






Uniqueness, keys, and foreign keys

■ Uniqueness and keys

□ Unique column: Only unique values

□ Unique column combination: Only unique value combinations

◊ Minimality: No subset is unique

□ Key candidate: No null values

◊ Uniqueness and non-null in one instance does not imply key: Only human can specify keys (and foreign keys)

■ Inclusion dependencies and foreign keys

□ A B: All values in A are also present in B

□ A1,…,Ai B1,…,Bi: All value comb. in A1,…,Ai are also present in B1,…,Bi

□ Prerequisite for foreign key

◊ Across relations and across databases

◊ Again: Discovery on a given instance, only user can specify for schema


31

Uniqueness and keys

■ Unique column

□ Only unique values

■ Unique column combination

□ Only unique value combinations

□ Minimality: No subset is unique

■ Uniques: {A, AB, AC, BC, ABC}

■ Minimal uniques: {A, BC}

■ (Maximal) Non-uniques: {B, C} Felix Naumann | Data Profiling | CUSO 2014

32

A B C

a 1 x

b 2 x

c 2 y

Null values

■ Null values have a wide range of interpretations.

□ Unknown (date of birth)

□ Non-applicable (driver license number for kids)

□ Undefined (result of integration/outer join)

■ What are minimal uniques for the following data set?

■ Primary key {A}; Some unusual uniques: {C} and {CD}

■ Distinct: {A, BC} but not {CD}


33

A B C D

a 1 x 1

b 2 y 2

c 3 z 5

d 3 ⊥ 5

e ⊥ ⊥ 5

Pruning effect of a pair


34

A B C D E

AB AC AD AE BC BD BE CD CE DE

ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

ABCD ABCE ABDE ACDE BCDE

ABCDE minimal unique

unique

Pruning with uniques

■ Pruning: inferring the type of a combination without actual

verification

■ If A is unique, supersets must be unique

■ Finding a unique column prunes half of the lattice

□ Remove column from initial data set and restart

■ Finding a unique column pair removes a quarter of the lattice

□ In general, the lattice over the combination is removed

■ The pruning power of a combination is reduced by prior findings

□ AB prunes a quarter

□ BC additionally prunes only one eighth

□ ABC already pruned one eights


35

Pruning both ways


36

A B C D E




ABCDE minimal unique

unique

maximal non-unique

non-unique

TPCH – Uniques and Non-Uniques


37 non-unique unique

8 columns

9 columns

10 columns

Unique Column Combination Discovery

■ DUCC

□ Basic idea: random walk through lattice

□ Pick random superset if current combination is non-unique

□ Pick random subset otherwise

□ Lazy prune with previously visited nodes


38

Row-based Column-based Hybrid

Gordian

[SBHR06]

Apriori

[GW99]

HCA

[AN11]

DUCC

[HQA+14]

SWAN

[AQN14]

A B C D E




ABCDE

ABCD

ABC

ABCE

ABD

ABDE

AB

ACD

CD

ACD BCD CDE

Minimum unique column combination candidate

Minimum unique column combination

Maximum non-unique column combination candidate Maximum non-unique column combination

Pruned

Visited nodes: 10 out of 26


39

http://openclipart.org/detail/1645/cancel-by-dagobert83






















Scaling the number of columns

■ NCVoter, 100k rows


40

Scaling the number of rows

■ NCVoter, 15 columns


41

Analysis of DUCC

■ Runtime mainly depends on size of solution set

■ Worst case: solution set in the middle


42

Uniques and non-uniques in NC-voter data

■ A minimal unique: voter_reg_num, zip_code, race_code

■ A maximal non-unique: voter_reg_num, status_cd, voter_status_desc, reason_cd, voter_status_reason_desc, absent_ind, name_prefx_cd, name_sufx_cd, half_code, street_dir, street_type_cd, street_sufx_cd, unit_designator, unit_num, state_cd, mail_addr2, mail_addr3, mail_addr4, mail_state, area_cd, phone_num, full_phone_number, drivers_lic, race_code, race_desc, ethnic_code, ethnic_desc, party_cd, party_desc, sex_code, sex, birth_place, precinct_abbrv, precinct_desc, municipality_abbrv, municipality_desc, ward_abbrv, ward_desc, cong_dist_abbrv, cong_dist_desc, super_court_abbrv, super_court_desc, judic_dist_abbrv, judic_dist_desc, nc_senate_abbrv, nc_senate_desc, nc_house_abbrv, nc_house_desc, county_commiss_abbrv, county_commiss_desc, township_abbrv, township_desc, school_dist_abbrv, school_dist_desc, fire_dist_abbrv, fire_dist_desc, water_dist_abbrv, water_dist_desc, sewer_dist_abbrv, sewer_dist_desc, sanit_dist_abbrv, sanit_dist_desc, rescue_dist_abbrv, rescue_dist_desc, munic_dist_abbrv, munic_dist_desc, dist_1_abbrv, dist_1_desc, dist_2_abbrv, dist_2_desc, confidential_ind, age, vtd_abbrv, vtd_desc


43

Dynamic Data: Challenges

■ Inserts may create new duplicate combinations

□ Minimal uniques (mUCs) might become non-unique

□ Maximal non-uniques (mNUCs) might lose maximality

■ Deletes remove duplicate value combinations

□ NUCs might get unique

□ mUCs might lose minimality

■ Idea

□ Leverage the knowledge of previously discovered mUCs and

mNUCs

□ Create appropriate indices


44

SWAN Architecture [AQN14]


45

SW AN

Database(input dataset)

Repository(MUCS and MNUCS)

Inserts Handler

Uniqueness

Checker

Deletes Handler

Duplicate

Checker

deletesinserts

MUCS-indexdata-index duplicate-index

inserts/deletes

inserts/deletes

update

Scaling the Number of Columns

■ 100k rows and 10k inserts


46

0.2$ 0.9$1$

10$

100$

1000$

10000$

100000$

10$ 20$ 30$ 40$ 50$ 60$

Ex

ec

uti

on

tim

e (

s)

Number of columns

Ducc Gordian-Inc Swan

■ TPCH with 16 columns and 5 million rows

■ Swan/Ducc combination is able to process larger datasets than

Ducc on a static dataset

Stressing the Number of Inserts


47

0"

2000"

4000"

6000"

8000"

10000"

12000"

10%" 20%" 30%" 40%" 50%" 60%" 70%" 80%" 90%" 100%"

Ex

ecu

tio

n t

ime

(s)

Insert size wrt. initial dataset size

Ducc Swan

Next steps

■ Finding primary keys

□ Uniqueness is necessary criteria

□ No null values

□ Include other features

◊ Name includes “id”, number of columns

■ Partial uniques

□ 99.9% of the data unique

□ Useful to detect data errors

□ Gordian, HCA, and DUCC can be easily modified

■ Incremental discovery


48

Inclusion Dependencies: Definition

■ INDs involve more than one relation.

■ Let D be a relational schema and let I be an instance of D.

■ R[A1, …, An] denotes projection of I on attributes A1, … An, of

relation R: R[A1, …, An] = πA1, …, An(R)

■ IND = R[A1, …, An] S[B1, …, Bn], where R, S are (possibly

identical) relations of D.

□ Projection on R and S must have same number of attributes.

■ An instance I of D satisfies if I(R)[A1, …, An] I(S)[B1, …, Bn]

■ Values of R: “dependent values”

■ Values of S: “referenced values”


49

IND types

■ Unary INDs

□ INDs on single attributes: R[A] S[B]

■ n-ary INDs

□ INDs on multiple attributes: R[X] S[Y]

■ Partial INDs

□ IND R[A] S[B] is satisfied for x% of all tuples in R

□ IND R[A] S[B] is satisfied for all but x tuples in R

■ Approximate INDs

□ IND R[A] S[B] is satisfied with probability p.

□ Based on sampling or other heuristics


50

Motivation for IND discovery

■ General insight into data

■ Detect unknown foreign keys

■ Example

□ PDB: Protein Data Bank

□ OpenMMS provides relational schema

◊ Parses protein and nucleic acid

macromolecular structure data

from the standard mmCIF format.

□ 175 tables with primary key

constraints

□ 2705 attributes

□ But: Not a single foreign key

constraint!


51

Motivation for IND discovery

■ Ensembl – genome database

□ shipped as MySQL dump files

□ more than 200 tables

□ Not a single foreign key constraint!

■ Why are FKs missing?

□ Lack of support for checking foreign key constraints in the

host system

◊ Example: Oracle did not support FKs up to v6

□ Fear that checking such constraints would impede database

performance

□ Lack of database knowledge within the development team


52


53

SPIDER: Single Pass Inclusion DEpendency

Recognition [BLNT07]

■ Main ideas

□ Test all IND-candidate pairs in parallel.

□ Read attribute values only once.

□ Stop test of an IND-candidate after first counter-example.

□ Reduce number of value comparisons by specialized data structure.

□ No need to build inverted index.

■ Two steps:

□ Sort and distinct all attribute‘s values and write them to disk

◊ For each attribute: SELECT DISTINCT A FROM R ORDER BY A

□ Test all IND candidate pairs in parallel

SPIDER by example

■ In each step: Intersect „attributes to process“ with each refs list of

previous step


54

attributes A, B, C

A B C

s s

t t t

x

y y y

z

attributes

to process

dep A

refs

dep B

refs

dep C

refs

Init B,C A,C A,B

Step 1 A,C C A,C A

Step 2 A,B,C C A,C A

Step 3 A A,C A

Step 4 A,B,C A,C A

Step 5 C A,C

Problem: Automatic Determination of Foreign Keys

■ Given

□ Relational schema

□ Database instance of that schema

□ Complete set of (observed) inclusion dependencies

◊ Attributes A and B with R[A] S[B] (in short A B)

■ Find

□ All foreign key constraints: attributes A and B with A B

■ Difficulty

□ Foreign keys are not intrinsic to data, but defined by humans

□ Discover semantics

■ Machine learning approach based on syntactic features [RAB+09]


55

Features

■ DependentAndReferenced

□ Counts how often the dependent attribute A appears as referenced attribute in the set of all INDs.

□ Usually, a foreign key is not also a primary key that is referenced as foreign key by other tables.

■ MultiDependent

□ Counts how often A appears as dependent attribute in the set of all INDs.

□ If s(A) is contained in the set of values of many other attributes, the likelihood for each of these INDs being a FK is decreased.

■ MultiReferenced

□ Counts how often B appears as referenced attribute in the set of all INDs.

□ Often, primary keys are referenced by more than one foreign key.


56

A

a

B

a

b

?

C

a

D

a

A

a

B

a

b

?

C

a

D

a

A

a

B

a

b

?

C

a

D

a

Features

■ DistinctDependentValues

□ The cardinality of s(A).

□ Usually, attributes that are foreign keys

contain at least some different values.

■ ValueLengthDiff

□ Difference between the average value length

(as string) in s(A) and s(B).

□ Usually, average length of the values is similar

whenever foreign keys reference a non-biased

sample of the primary keys.


57

A

a

a

a

a

a

B

a

b

c

d

e

?

A

abab

abab

abab

c

d

B

abab

b

c

d

e

?

Features

■ Coverage

□ The ratio of values in s(B) that are covered by s(A)

compared to all values in s(B).

□ Usually, foreign keys cover a considerable number of

primary key values.

◊ 60% of FK-attribute values cover all ref-values

◊ Each covers at least 10%

■ OutOfRange

□ Percentage of values in s(B) that are not within

[ min(s(A)), max(s(A)) ].

□ Usually, the dependent values should be evenly

distributed over the referenced values.

□ Mostly, less than 5% of values outside of range

■ TableSizeRatio

□ Ratio of number of tuples in A and number of tuples in B.

□ Usually in life sciences databases, table sizes do not

differ wildly


58

A

b

c

b

c

B

a

b

c

d

e

f

g

?

Features

■ ColumnName

□ Similarity between name(A) and

name(B), also considering the

name of the table of which B is

an attribute.

■ TypicalNameSuffix

□ Checks whether name(A) ends

with a substring that indicates a

foreign key.

□ „id“, „key“, and „nr“


59

FILMTEXTE.FILMTEXTTYPNR

FILMTEXTTYPEN.FILMTEXTTYPNR

CUSTOMER.C_NATIONKEY

NATION.N_NATIONKEY

SG_SEQFEATURE.ENT_OID

SG_COMMENT.ENT_OID

COURSE.STUDENT

STUDENT.ID

SG_BIOENTRY.TAX_OID

SG_TAXON.OID

Agenda

60







Tools have very long feature lists


61

■ Num rows

■ Min value length

■ Median value length

■ Max value length

■ Avg value length

■ Precision of numeric values

■ Scale of numeric values

■ Quartiles

■ Basic data types

■ Num distinct values ("cardinality")

■ Percentage null values

■ Data class and data type

■ Uniqueness and constancy

■ Single-column frequency histogram

■ Multi-column frequency histogram

■ Pattern discovery (Aa9)

■ Soundex frequencies

■ Benford Law Frequency

■ Single column primary key discovery

■ Multi-column primary key discovery

■ Single column IND discovery

■ Inclusion percentage

■ Single-column FK discovery

■ Multi-column IND discovery

■ Multi-column FK discovery

■ Value overlap (cross domain analysis)

■ Single-column FD discovery

■ Multi-column FD discovery

■ Text profiling

Oracle Data Profiling and Quality Control Center


62

Screenshots from IBM Information Analyzer


63

Typical Shortcomings of Tools (and methods from research)

■ Usability

□ Complex to configure

□ Results complex to view and interpret

■ Scalability

□ Main-memory based

□ SQL based

■ Efficiency

□ Coffee, Lunch, Overnight

■ Functionality

□ Restricted to simplest tasks

□ Restricted to individual columns or small column sets

◊ “Realistic” key candidates vs. further use-cases

□ „Checking“ vs. „discovery“

■ Interpretation of profiling results


64

That‘s the big one

Metanome – Profiling your Datanome


65

Algorithm execution Result

management

Algorithm configuration Result

presentation

Configuration

Measurements SPIDER

jar

DUCC jar

SWAN jar

txt

xml csv

DB2 DB2

MySQL

Results

Agenda

66







Online Profiling

■ Profiling is long procedure

□ Boring for developers

□ Expensive for machines (I/O and CPU)

■ Challenge: Display intermediate results

□ … of improving/converging accuracy

□ Allows early abort of profiling run

■ Gear algorithms toward that goal

□ Allow intermediate output

□ Enable early output: “progressive” profiling


67

Incremental Profiling

■ Data is dynamic

□ Insert (batch or tuple-based)

□ Updates

□ Deletes

■ Problem: Keep profiling results up-to-date…

□ … without re-profiling the entire data set.

□ Easy examples: SUM, MIN, MAX, COUNT, AVG

□ Difficult examples: MEDIAN, uniqueness (see earlier slides),

dependencies


68

Piggyback Profiling

■ Goal: Determine metadata for query results

■ Challenge: With as little query processing overhead as possible

□ Baseline: Run second SQL query

□ Piggybacking: profile along query plan (using base statistics)


69

Profiling for Integration

■ Profile multiple sources simultaneously

■ Schema matching/mapping

□ What constitutes the “difficulty” of matching/mapping?

■ Duplicate detection

□ Estimate data overlap

□ Estimate fusion effort

■ Create measures to estimate

integration (and cleansing) effort

□ Schema and data overlap

□ Severity of heterogeneity


70

Profiling new Types of Data

■ Traditional data profiling: Single table or multiple tables

■ More and more data in other models

□ XML / nested relational / JSON

□ RDF triples

□ Textual data: Blogs, Tweets, News

□ Multimedia data

■ Different models offer new dimensions to profile

□ XML: Nestedness, measures at different nesting levels

□ RDF: Graph structure, in- and outdegrees

□ Multimedia: Color, video-length, volume, etc.

□ Text: Sentiment, sentence structure, complexity, and other

linguistic measures


71

Average Sentence Length


75

„Literature Fingerprinting: A New Method for Visual Literary Analysis” by Daniel A. Keim and Daniela Oelke

Hapax Legomena


76

„Literature Fingerprinting: A New Method for Visual Literary Analysis” by Daniel A. Keim and Daniela Oelke

News Statistics


77

Master thesis Matthias Kohnen

Summary

78







Summary


79

Data Profiling

Single source

Single column

Cardinalities

Uniqueness and keys

Patterns and data types

Distributions

Multiple columns

Uniqueness and keys

Inclusion and foreign key

dep.

Functional dependencies

Conditional and approximate

dep.

Multiple sources

Topical overlap

Topic discovery

Topical clustering

Schematic overlap

Schema matching

Cross-schema dependencies

Data overlap

Duplicate detection

Record linkage