DATA - Would you like Quality with that? Pat McKeefry April 2009 15/10/2015© Diffusion Limited 20091

DATA - Would you like Quality with that?

Pat McKeefryApril 2009

20/04/23 © Diffusion Limited 2009 1

Data - The Intangible Asset

• A companies Data defines who and what a company really is......

- not branding or marketing- but in a DNA type context, i.e. what really makes the company

tick

• This information includes;- Client data- Internal process data- Customer accounting data- Customer Relationship data- Marketing material- Correspondence- and much, much more


Data - The Intangible Asset ..

• And all this information is used to support every aspect of the companies activities;

- for Financial management- for Compliance reporting- for Shareholder reporting- for Management decision making- for Sales Planning- for Marketing Planning- for Production Planning- for Procurement Planning- etc, etc


Data - The Intangible Asset ...

• So, do we think Data is important?- not merely important, Critical.

• “70 percent of small firms that experience a major data loss go out of business within a year.” - Contingency Planning, Strategic Research Corp US 2007

“93% of companies that lost their data centre for 10 days or more due to adisaster filed for bankruptcy within one year of the disaster.” - National Archives & Records Administration in Washington 2007

Without Data, there is no business...


Data - The Intangible Asset ....

• If data is critical for a company to survive, does it follow that the data should also be: correct and up to date?

- Absolutely

What happens if it isn’t.....

- rework- increased costs- loss of revenue- unreliable information- possible penalties- otherIndirectly, poor quality data hits the bottom line.

The case for Data Quality


So what is Data Quality?

• For many companies, it is an elusive dream.....Why?

Because most companies treat data quality as a single event- one off initiatives to ‘Cleanse data’- one off initiatives to ‘fill data gaps’- one off initiatives to ‘populate missing data’- one off initiatives to ‘rationalise the customer database’- and so on ...


So what is Data Quality?..

Data Quality is a process, not an event!


What do we get from High Quality Data - Benefits

• At a Local level- reduction of problems within applications- consistent, accurate, and reliable data (but silo’ed)

• At an Enterprise level- consistent, accurate, and reliable data (integrated)- unified view of the enterprise- effective strategies- better performance- better productivity


Data and the SDLC – Typically No detection of Data Issues

• Business case/planning– forgetting to include or make sufficient estimate for the data cleansing

and data migration effort

• Process Design/Blueprint– BA analyses as-is and designs for to-be processes, but typically does

not look at data in legacy systems; many processes have built in rules (undocumented) for dealing with data combinations

– Even data driven designs tend to do data modelling without looking at the data content of source systems


Data and the SDLC – Issues start to appear…

• Build phase– Programmers build to specs– Typically programmers test with mock up data to unit test the new

programs

• Function/Integration Test– Test Analyst writes Test Scripts for Design specs, which typically don’t

address data or simple pre-requisites describe what the tester needs to get to test the scenarios.

– This leaves the tester with a problem as to how to get the data, make it up or get a snapshot of live data? Depending on approach this is where the problems appear


The Start up position – Data Migration

• There’s a saying that the two most common causes of project overruns / failures are:

Politics and Data Migration


The Start up position – Data Migration

• It seems obvious, but what is Data Migration?

“Data Migration is the reconciliation of what data you think you need in your new system, with what data you actually need .........loading is relatively straight forward!” Tom Kennedy - BackOffice Associates


The Start up position – Data Migration: Key Points

• Identification of where the data is coming from to populate the new system

- single legacy system- multiple legacy systems

- versions of data- concatenation of data

• A very good understanding of what sort of state this data is in- structure (drives mapping)- quality (missing, invalid, duplicated, incorrect)

• An understanding of the data structures in the new system- structure (transformations)


The Start up position – Data Migration: Key Points

• In practice identification of source systems & understanding of the new systems data structures are relatively easy.

• The issue lays with understanding the state of the legacy data

This is because:- most companies do not know what state their data is in, good or bad!


The Start up position – Data Migration & Data Quality

• Often the migration of Data to a new system is the first step towards Data Quality

- usually data is verified as ‘fit for purpose’, i.e. only what is needed to support the new system is migrated, and it

has gone through a cleansing process (a fresh start)


The Start up position – Data Migration: Tips

• Treat data migration as a sub project in its own right- budget (be realistic)- resource (not an army, but skilled experts)- tools

• Data Migration is a business process, not a technical problem- get key business people involved early- don’t underestimate the effort required to reach an

acceptable quality level to load

• Use a toolset- automate as much as possible- define business rules; data cleansing, transformation


Maintaining Data Quality

• Assuming you are able to start the new system on a good Data Quality footing

- cleansed data- successfully migrated- all verified and reconciled

How do you go about keeping it that way?

Dunn & Bradstreet reports its database of businesses experience annual changes of 20% for addresses, 17% for business names, and 18% for phone numbers.



• Remember that Data Quality is a process, not an event• Management of data quality should sit within an overarching

Master Data Management framework that incorporates:

- Quality- Governance (Data Architecture)

- Stewardship


Maintaining Data Quality – a Framework

• A Master Data Management framework consists of:

- People (executive sponsorship, business & IT commitment)

- Policies (creation and enforcement of policies that guide the collection and management of data)

- Technology (collaboration tools that support an enterprise view of data)


Maintaining Data Quality - Governance

• Data Governance :- identifies the data objects and elements to be managed

- specifies the policies and business rules* for how master data is created and maintained

- describes any hierarchies, taxonomies, or other relationships needed to organise or classify the objects

- assigns data stewardship responsibility to individuals within the company

* Data quality can be measured against business rules (typically 2 – 300)



• Start by reviewing your Data Governance Maturity

Level 1 – Undisciplined- limited view of data quality problems

Level 2 – Reactive- begin to understand the role of data governance

Level 3 – Proactive- benefits establish foundation for MDM and business process automation

Level 4 – Governed- enterprise wide data quality as integral part of business processes


Maintaining Data Quality – Level 1 Governance

Characteristics of an Undisciplined companyPeople Success depends on the competence of a few individuals BA’s are removed from the development of business rules No management input or buy-in on data quality problems Executives are unaware of data problems or blame IT entirely Company relies on personnel that follow different paths to reconcile and correct data

Policies Data Quality is non-existent or project focused only, with no defined data quality processes Data and data processing is siloed – systems operate independently ‘Fire fighting mode’. Address problems as they arise through manually driven processes Company resources are not optimal due to redundant outdated data

Technology No data profiling, analysis or auditing is used Data standardisation and cleansing only occurs in isolated systems Data improvements are focused on single applications, e.g. database marketing, or sales force automation

Risk and Reward Risk: Extremely high. Data problems result in lost customers or improper procedures. A few scapegoats receive the blame, although it is impossible to

accurately assign culpability Reward: low. Outside the success of a few employee’s, companies reap few benefits from data quality



Characteristics of a Reactive companyPeople• Success depends on a group of database administrators or other employee’s• Individuals create useful processes for data quality initiatives, but no standard procedures exist across functional areas• Little corporate management buy-in to the value of data or to an enterprise wide approach to data quality or data

integration Policies• Rules for data governance emerge, but the emphasis remains on correcting data issues as they occur• Most data management processes are short range and focus on recently discovered problems• Within individual groups and departments / units, tasks and roles are standardised Technology• Tactical data quality tools are often available; data profiling, data quality• Application like CRM and ERP utilise data quality technology• Most data is not integrated across business units, but some departments attempt isolated integration efforts• Database administration tactics emerge, e.g. reactive performance monitoringRisk and Reward• Risk: High, due to a lack of data integration and overall inconsistency of data throughout the enterprise. While data is

analysed and corrected sporadically, data failures can still occur on a cross functional level• Reward: Limited and mostly anecdotal. Most return on investment comes via individual processes or individuals, and there

is limited corporate wide recognition of data quality benefits



Characteristics of a Proactive companyPeople• Management understands and appreciates the role of data governance and commits resources• Executive level decision makers begin to view data as a strategic asset• Data stewards emerge as the primary implementers of data management strategies and work directly with cross functional

teams to enact data quality standardsPolicies• Real time activities and preventative quality rules and processes emerge• Data governance processes are built into customer, product and other solutions• Data metrics are sometimes measured against industry standards to highlight areas for improvement• Goals shift from problem resolution to preventionTechnology• A data stewardship group maintains corporate data definitions and business rules• Service orientated architecture becomes the enterprise standard• Ongoing data monitoring helps the company maintain data integrity• More real time processes is available and data quality functionality is shared across different operations modes Risk and Reward• Risk: Medium to low. Risks are reduced by providing better information to increase the reliability of sound decision making• Reward: Medium to high. Data quality improves, often in certain functional areas and then in a wider context as more

employee’s join the early adopters



Characteristics of a Governed companyPeople• Data governance has executive level sponsorship with direct CEO support• Business users take an active role in data strategy and delivery• A data quality or governance group works directly with data stewards, application developers, and database administrators• The company has ‘zero defects’ policies for data collection and managementPolicies• New initiatives are only approved after careful consideration of how they will impact the existing data infrastructure• Automated policies are in place to ensure that data remains consistent, accurate, and reliable throughout the enterprise• A service orientated architecture encapsulates business rules for data qualityTechnology• Data quality and data integration tools are standardised across the company• All aspects of the company use standardised business rule created and maintained by designated data stewards• Data is continuously monitored, with any deviations from standards resolved immediately• Data models capture the business meaning and technical details of all corporate data elementsRisk and Reward• Risk: Low. Master data is tightly controlled across the enterprise, allowing companies to maintain high quality information

about its customers, prospects, inventory, and products• Reward: High. Corporate data practices lead to a better understanding of the companies current business position, allowing

full confidence in all data based management decisions


Maintaining Data Quality – Advancing to the next stage

• Level 1:Undisciplined to Level 2:Reactive- usually triggered by an event or series of events related to poor data quality- recognition of data integrity problems- quantifying the effects of poor data quality in the company- when recognition motivates change, the company can reach the next level

• Level 2:Reactive to Level 3:Proactive

- big culture shift needed; control moves from local units / applications, to enterprise governance- high degree of executive commitment required- focus is initially on major data object(s); customers, products

• Level 3:Proactive to Level 4:Governed

- emergence of a ‘centre of excellence’ or similar framework with multiple data stewards- BA’s start to control data management processes- IT playing a supporting role


Maintaining Data Quality – Master Data Management Framework


Suppliers

Da

ta R

eg

istr

ies

Ind

ust

ry D

ata

R

ep

osito

rie

sT

radi

ng

P

artn

ers

Da

ta C

lea

nsin

g

Ser

vice

s

Customers

Suppliers

Assets

Employee's

Products

Stock

Stewardship & Business Rules

Taxonomy & Relationships

Master Data Architecture

Ma

ste

r D

ata

R

ep

osito

rie

s

Ma

p &

Mo

ve

ValidateCleanse & Enhance

Secure

Workflows

Master Data ManagementComponents and Services

Master Data Governance

Unstructured Data

ContentTransaction Applications Analytics

Reporting

SRM ERP

PLM SCM

CRM

EAM

Maintaining Data Quality - Master Data Management Organisation


Data Quality – Yes please!

Thank you


Documents

DATA - Would you like Quality with that? Pat McKeefry April 2009 15/10/2015© Diffusion Limited 20091