Data Quality: Not Your Typical Database Problem

Data QualityNot your Typical Database Problem

2011 © Copyright QCRI. Confidential document.

Not your Typical Database Problem

Ahmed Elmagarmid

Executive DirectorQatar Computing Research Institute

Where are we located?

2011 © Copyright QCRI. Confidential document. 33

Qatar Foundation

EDUCATIONSCIENCE &RESEARCH

COMMUNITY DEVELOPMENT

2.8 percent of GDP to be spent on research

be spent on research annually by 2015

Biomedical

Qatar Energy &

Environment

Computing

Qatar Foundation Research Division

Biomedical

Research

Institute

Environment

Research

Institute

Computing

Research

Institute

QCRI Overview

QCRI Vision

To make Qatar a global center forcomputing research by becoming theworld’s recognized leader in Arabic

world’s recognized leader in Arabiclanguage technologies and in key areasvital to the global growth of Qataribusiness and entrepreneurial activity .

AcademiaAcademia

National Institutions (QCRI)

� Grand practical challenges � National and global impact� Localized skills & knowledge

National Institutions (QCRI)

� Grand practical challenges � National and global impact� Localized skills & knowledge

ngesQCRI Model

� Individual projects� Students move on� Theoretical & basic

research

� Individual projects� Students move on� Theoretical & basic

research

� Localized skills & knowledge� Large teams and long term� Example peers: INRIA, MPI

� Localized skills & knowledge� Large teams and long term� Example peers: INRIA, MPI G

Basic Research Applied Research

Research Parks

� Commercialization� Entrepreneurship� Incubation

Research Parks

� Commercialization� Entrepreneurship� Incubation

QCRI Ecosystem

QCRIQCRI

HKUHKUQEERIQEERI

QBRIQBRISidraSidra MITMIT

BoeingBoeing

AljazeeraAljazeera

YahooYahooGoogleGoogle

MicrosoftMicrosoft

ALTISALTIS

QSAQSAMEEZAMEEZA

QSTPQSTP

Energy Co.

WikiMediaWikiMedia

IBMIBM

Arabic Language

Social Computing

Scientific Computing

QCRI Research Centers

Language Technologies

Computing Computing

Data Analytics

Cloud Computing

Prof. Rich DeMilloGeorgia Tech, Chair

QCRI Scientific Advisory Council

Prof. Joichi Ito MIT Media Lab Director

Prof. Ruzena BajcsyUniversity of California – Berkeley

Lord Rupert RedesdaleUK House of Lords

Lew TuckerVice President, Cisco

Prof. Alfred V. AhoColumbia University

Yousef KhalidiVice President, Microsoft

Prof. Dick LiptonGeorgia Tech

Rashid

KamalHalima

Kulood

The 60 Doers!

Scientific

Ihab Mourad

Michele

ElKindi

Data AnalyticsPaolo

Management and Support Team

AhmedAbdellatif

Agathe

Melissa

Samreen

Computing

Richard P.

Richard

Simon G.

MohamedSimon P.

William

Khulood

Ahmed A.

Mustafa

Cloud Computing

Arabic Language

Technologies

Kareem Stephan

Preslav

Ahmed T.

Francisco

ThuyLinh

Safdar

AyshaSofiane

MikalaiRuth

Gautam

Social Computing

Samreen

Othmane

Ahmed A.

Aybuke Shameem

Walid Peng

Ahmed M.

Ahmed T.

Khaled

Strategic Partnerships

AgendaStrategic Partnerships

5-YEAR QCRI MANPOWER PLAN

10-11 11-12 12-13 13-14 14-15

34+13 +48 +20 +8

This Talk

Data Quality

Enhancing the usability of the acquired data and increasing the confidence of query results"Poor data quality is the norm rather than the exception, but most organizations are in a state of denial about this issue. " -Gartner Group

state of denial about this issue. " -Gartner Group

Real life data is often dirty: Data

error rates in industry: 1% - 30%

(Redman, 1998)

Dirty Data is Expensive

Obama administration offered

$19 billion grants for health IT, i.e.

improve EMRs in 2009

Erroneously priced data in retail

databases costs US customers

$2.5 billion each year

The Data Warehousing Institute

estimates that data quality

problems cost U.S. businesses

more than $600 billion a year

(2002)

Where to start? Data Quality everywhere !

• Data Entry• Information Extraction• Integration from multiple sources

• Integration from multiple sources• Standardization and transformation• Business rules compliance

““““Academic ”””” Data Cleaning

● Pick a well understood data problem under some scoping

assumptions and solve independently

� Duplicates

� Functional Dependency violations

� Matching dependency violations

� Missing value imputation

● Piece-meal approach to tackle the complexity and sometimes the

intractability of the problem

� Repairing violations of FD constraints in special cases (no deletion, left hand

side changes only, allowing variable etc.)

““““Academic ”””” Data Cleaning

• Despite their theoretic and algorithmic beauty, rarely used

– Problems never exist in isolation

– Fixes to one problem often introduce “other” problems– Data usually not accessible to mess with

– Data usually not accessible to mess with– Integrity constraints!... What integrity constraints?!!

““““Practitioner ”””” Data Cleaning

• Will share some scary stories

– “post-it notes” as an expert messaging system– “written permission” to change value of a record– Default values and best practices

– Default values and best practices

– “Call John.. He will know what to do”

This Talk

● Few data quality challenges and (hopefully) research

directions

● Summary of recent efforts at QCRI

10 Data Quality Issues

Issue 1: The data trio

Quality

Extraction remains a key source of data errors

Acquiring the semantics/schema of the underlying un structured data sources (document, emails, related Web info, click traces, profiles, interests, etc.)

Integration aggravates the problem

Linked data as an attempt to live with errors .. link as you go

m1 I'm not sure about this idea of putting "linked data" so prominent in this slide on IImourad, 7/23/2011

Issue 2: Data level or application level

• Cleaning data tables by trusting the schema table! Is rarely useful• Will share a story

– Bell-core with 1800 inter-linked databases– Rule-based logic for sanity checking– Post-it messages to communicate between data quality officers

– Post-it messages to communicate between data quality officers .. Who work in shifts!

– Data cleaning action is meaningless if not tied to a business logic or to a process. Should never be against FDs

Issue 3: Protect your gain: DQ Dashboard

● How to protect against going backwards

● How to protect your gains during the cleansing process

● Metrics:

�Minimality Principle: mostly and widely used in academic

cleaning

�Value of information: to spot the most important problem to fix

Issue 3: Protect your gain - Ideas

• Root-cause analysis for data cleaning

• Chase problems to the source to reason about “progress”

• Leveraging “Provenance” to design progress meters

Issue 4: Data is not an orphan!

● Data Stewards are not imaginary characters! Important data

has stewards and custodians

● Need to go through these guardians first

� Some health care requires a signed form per changed cell stating

reasons for change

● Possible approaches:

� How to avoid stewards?

� How to integrate them in the process or minimize their involvement?

Issue 5: How clean is clean?

• Quality awareness eats up 10% of the budget [Telecom Experience]

• How to avoid over-cleaning

• Example: “Bill Forgiveness”, a real-life experience: roaming

• Example: “Bill Forgiveness”, a real-life experience: roaming charges and cross-carrier calls have a very complicated business model

• Possible approaches

– Measure cleaning progress

– Clean only to satisfy some application needs

Issue 6: Online cleaning a necessity not a feature

● We live in a complex world → complex applications with 100s and 1000s of components and parameters

● Clean as you go .. Clean on demand .. Clean opportunistically .. Can be the only hope

● New concepts:� Iterative cleaning

� Cleaning dynamic and evolving data

● Off-line cleaning can still benefit historical data but is becoming less and less important

Issue 7: Application quality

• Data Quality → Information Quality → Application quality

• Realizes the levels of complexity in current BI apps

• Data usage should influence data cleaning

– “Usage-based” data cleaning

Issue 8: SW engineering DQ

• Current focus on discrete values with simple integrity constraints (FD, uniqueness…)

• We are good at checking if data complies with rules

• Real business rules are often “assertions” and expressed in

• Real business rules are often “assertions” and expressed in “turing-complete” languages

• Checking “did we write the assertions right?” becomes a lot harder

• But also.. need to think if we wrote the right assertions!

Issue 9: DQ Theory?

• ACID in transaction management were not only sensible requirements but also had algorithms and methods to enforce them during transactions processing

• Does it make sense to do the same for Quality? Plausible properties along

• Does it make sense to do the same for Quality? Plausible properties along with actions for maintaining acceptable quality during data manipulation

• Some of these already exist: Timeliness, Currency, Consistency, etc. but lack methods of enforcement

Issue 10: Scale .. Scale

• Terabytes and Petabytes of data requires new ways to enforce data quality

• Which ball to drop

• Leveraging application semantics and data usage

• Sampling to learn from the few and apply on the masses

• Active learning to replace human feedback (GDR as a solution)

Sample QCRI Projects

GDR – Guided Data Repair

• Scalable ways to involve experts

• Repurposing destructive automatic techniques to guide repairs

• Value of Information measures to generate the most important

questions

• Judicious use of active learning from user feedbackUser QueryUser Query

• Judicious use of active learning from user feedback

Input Database

Instance

Detect Errors

and Violations

Learn and

Repair

Database

Clean Database

Instance

User QueryUser Query

Results

GDR Architecture

Probabilistic Data Cleaning

Uncertain

Error Detection

Possible

Repair

Generation

Clean Database

Instance

User QueryUser Query

Clean Database

InstancePossible

Database

Instance

Probabilistic Results

InstancePossible

Clean Instance

Possible Repairs

A possible repair is a clustering of the input tuples

Person

{P1,P2}

{P1,P2,P5}

Possible RepairsID Name ZIP Income

P1 Green 51519 30k

Uncertain Clustering

{P3,P4}

{P1,P2}

{P3,P4}

{P1,P2,P5}

{P3,P4}

P1 Green 51519 30k

P2 Green 51518 32k

P3 Peter 30528 40k

P4 Peter 30528 40k

P5 Gree 51519 55k

P6 Chuck 51519 30k

Thank You

www.qcri.qa

Data Quality: Not Your Typical Database Problem

Technology

Edp typical problem

Database Concepts - MSDIS“database problem” rather than an algorithm, analysis, data input or data display problem 5 Definition A database is a collection of non-redundant data

Can the project be a database problem - Home | …web.engr.oregonstate.edu/~dambrobr/classes/cs461/project... · Web viewCan the project be a database problem? For example, I am the

Oracle Cost Based Optimizer (CBO) and Its Optimism About Cardinalities%3a Typical Problem Situations

Oracle Database Security - SF ISACA Database Security Paul Needham ... described for Oracle ... Typical Application Security with APEX Direct, Uncontrolled AccessPublished in: Information

Introductory Database Concepts - Jones & Bartlett Learning · Introductory Database Concepts ... When you buy goods in a supermarket or retail store ... Figure 1.3 shows a typical

Searching a Typical Database - University of Maine System

DATABASE MANAGEMENT SYSTEMS - Swarnandhra … … · · 2017-02-03Fundamentals of Relational Database Management Systems, S.Sumathi, S.Esakkirajan, ... source shortest path problem

On the database lookup problem of approximate … The Database Lookup Problem Of Approximate Matching By ... Indexing problem of similarity digests 1. ... On the database lookup problem

Web viewThe four steps to problem-solving are Understand, Plan, Solve and Check. 2. What are some strategies for problem-solving? Typical strategies for problem-solving are:

Fall Internship, 2015€¦ · Adding Nodes How to handle new types of hardware? Typical database method Redesign database NOSS method Nodes stored as XML in database Dynamically creates

Unit 11: Problem Management and Counseling Phases of Problem Management Typical Diving Problems And Solutions –Physical problems –Psychological problems

ADBIS 2007 Discretization Numbers for Multiple-Instances Problem in Relational Database

Polyinstantiation Problem CPSC 4670 Database Security and Auditing

SQL Server Database Development Best Practices€¦ · SQL Server Database Development Best Practices ... Build a Data Modeling Checklist for your company ... database design problem

Face Images Database Indexing for Person Identification Problem

ContentProviders. Databases for reading & writing data Support typical database operations e.g., query, insert, update & delete

The Inverse Problem of Seismic Velocitiestrip.rice.edu/downloads/siam_is.pdf · seismic wave motion (acoustic, elastic, viscoelastic,...) Typical problem size for adequately sampled

A Novel Method for the Absolute Pose Problem with Pairwise ... · Absolute pose estimation is a fundamental problem in computer vision, and it is a typical parameter estimation problem,

Database Security 101 - Universitetet i oslo · Test copy Production Storage Users WebServer AppServer Prod Database Backup Typical 3 tier architecure