Data Quality:Data Quality:Treasure in/Treasure OutTreasure in/Treasure Out

Victoria Essenmacher,  SPEC Associates Victoria Essenmacher,  SPEC Associates Melanie Hwalek, SPEC AssociatesMelanie Hwalek, SPEC Associates

Portions of this presentation were created in partnership with Portions of this presentation were created in partnership with Michel Lahti, University of Southern Maine (email: mlahti@usm.maine.edu)Michel Lahti, University of Southern Maine (email: mlahti@usm.maine.edu)

SPEC Associates, 615 Griswold St., Ste. 1505, Detroit, MI 48226SPEC Associates, 615 Griswold St., Ste. 1505, Detroit, MI 48226Phone: (313) 964-0500Phone: (313) 964-0500

Web site: www.specassociates.orgWeb site: www.specassociates.orgEmails: vessenmacher@specassociates.orgEmails: vessenmacher@specassociates.org

mhwalek@specassociates.orgmhwalek@specassociates.org

Presentation for the Michigan Nonprofit Association SuperConferencePresentation for the Michigan Nonprofit Association SuperConferenceMay 15, 2007May 15, 2007

1. Government Performance and 1. Government Performance and Results Act (1993) Results Act (1993)

Message: What gets measured gets doneMessage: What gets measured gets done

2. No Child Left Behind (2002) 2. No Child Left Behind (2002) Message: Data-based decision-making for Message: Data-based decision-making for

better or for worsebetter or for worse

3. Sarbanes-Oxley Act (2002)3. Sarbanes-Oxley Act (2002) Message: Everyone is accountable for Message: Everyone is accountable for

quality of data reportingquality of data reporting

4. Panel on the Non Profit Sector 4. Panel on the Non Profit Sector (2005) (2005)

Recommendation #5: Provide detailed Recommendation #5: Provide detailed information about programs, including information about programs, including methods used to evaluate program methods used to evaluate program outcomesoutcomes

What do we mean by internal What do we mean by internal quality assurance of data?quality assurance of data?

What do we mean by internal What do we mean by internal quality assurance of data?quality assurance of data?

The planning and implementation of practices The planning and implementation of practices that insure “adequate” and consistent quality that insure “adequate” and consistent quality related to the internal conduct of data planning, related to the internal conduct of data planning, collection, analysis and reporting, in order to:collection, analysis and reporting, in order to:

– Provide a system for team members to followProvide a system for team members to follow– Eliminate (or minimize) human errorEliminate (or minimize) human error– Provide “adequate” documentation of the Provide “adequate” documentation of the

program/project/organization program/project/organization

How are issues of internal quality How are issues of internal quality assurance addressed currently in assurance addressed currently in

the field of evaluation?the field of evaluation?

No broadly accepted guidelines or No broadly accepted guidelines or standards of data qualitystandards of data quality

Much variation - quality assurance is often Much variation - quality assurance is often particular to a person or organizationparticular to a person or organization

Can borrow from other fieldsCan borrow from other fields

Why might data quality be Why might data quality be important to address in a nonprofit important to address in a nonprofit

organization?organization?Have a known minimum accepted level of Have a known minimum accepted level of practice within your organizationpractice within your organization

Understand what data in reports actually Understand what data in reports actually mean and how they were derivedmean and how they were derived

Provide “real” information to base Provide “real” information to base decisionsdecisions

Today’s AgendaToday’s Agenda

Complete checklist: Organizational Capacity to Produce Complete checklist: Organizational Capacity to Produce Defensible Program DataDefensible Program Data

Overview of eight steps to data quality Overview of eight steps to data quality

Quick summary of results: Identifying common weak Quick summary of results: Identifying common weak areas areas

Exploration and examples for two common weak areasExploration and examples for two common weak areas

Wrap up Q&AWrap up Q&A

1

1

2

2

3

3

3

3

3

3

3

3

1. Planning for programdata collection

2. Beginning datacollection

3. Editing datacollection

4. Data entry andcleaning

5. Preparation of datafor analysis

6. Data analysis

7. Reporting

8. Data storage andsecurity

0 capacity 1 of 3 2 of 3 3 of 3

Eight components of data quality assurance

How does your organization score?How does your organization score?

1.1. Planning for program data Planning for program data collectioncollection

ScenarioScenario: A program manager did not do : A program manager did not do a good job of documenting. He had to a good job of documenting. He had to leave the program unexpectedly and the leave the program unexpectedly and the next person was left not knowing the next person was left not knowing the program design, and what program program design, and what program information was being collected or planned information was being collected or planned to be collected. Because it took so much to be collected. Because it took so much time to find all of the “pieces” the time to find all of the “pieces” the organization missed a reporting deadline organization missed a reporting deadline to a funder.to a funder.

2. Beginning data collection2. Beginning data collection

ScenarioScenario: In the rush of the moment, a decision : In the rush of the moment, a decision was made to add a follow-up survey to an was made to add a follow-up survey to an evaluation that involved pre and post testing. evaluation that involved pre and post testing. The new survey was sent to the program sites The new survey was sent to the program sites and a third round of data were collected on and a third round of data were collected on program participants. There was no time to program participants. There was no time to “proof” the survey prior to copying and “proof” the survey prior to copying and distributing. Inadvertently, the unique ID# for distributing. Inadvertently, the unique ID# for each participant was left off of the survey tool each participant was left off of the survey tool meaning that the follow-up data could not be meaning that the follow-up data could not be matched with the pre-post survey for proper data matched with the pre-post survey for proper data analyses. The time spent gathering the data was analyses. The time spent gathering the data was wasted.wasted.

3. Editing data collection3. Editing data collection

ScenarioScenario: A supervisor found it curious : A supervisor found it curious that a part-time telephone interviewer that a part-time telephone interviewer reported the exact same times on her reported the exact same times on her timesheet every day, as compared with timesheet every day, as compared with other part-time interviewers. Validation other part-time interviewers. Validation phone calls were made to persons phone calls were made to persons interviewed by this interviewer. It was interviewed by this interviewer. It was determined that the interviewer was not determined that the interviewer was not making the calls and fabricated the making the calls and fabricated the interview data. interview data.

4. Data entry and cleaning4. Data entry and cleaning

ScenarioScenario: 20 program sites emailed Excel files : 20 program sites emailed Excel files of enrollment and school report card data to the of enrollment and school report card data to the program manager. A support staff manually cut program manager. A support staff manually cut and pasted each site’s spreadsheet into a and pasted each site’s spreadsheet into a master file so that results could be compared master file so that results could be compared across programs. In looking at the master file, across programs. In looking at the master file, the evaluation manager found that the total the evaluation manager found that the total number of records in the file was 50 students number of records in the file was 50 students short of the total number of students that the short of the total number of students that the program sites reported serving. There was no program sites reported serving. There was no way to identify which 50 students were missing way to identify which 50 students were missing in the master file without repeating the cut and in the master file without repeating the cut and paste process. paste process.

5. Preparation of data for analysis5. Preparation of data for analysis

ScenarioScenario: A report from a community : A report from a community survey indicated that the average age of survey indicated that the average age of the urban community was 78. The the urban community was 78. The community had no nursing home or community had no nursing home or retirement community. In the database, retirement community. In the database, missing data was coded as 99.missing data was coded as 99.

6. Data analysis6. Data analysisScenarioScenario: An organization established an annual : An organization established an annual customer satisfaction survey. The first year’s results customer satisfaction survey. The first year’s results showed very high customer satisfaction which was showed very high customer satisfaction which was reported to the funder. Before the data from the reported to the funder. Before the data from the second survey was analyzed, the program staff who second survey was analyzed, the program staff who analyzed the first survey data left and a new staff analyzed the first survey data left and a new staff person was hired. The new staff person analyzed the person was hired. The new staff person analyzed the second year’s survey and found a dramatic decrease second year’s survey and found a dramatic decrease in the level of customer satisfaction compared to in the level of customer satisfaction compared to year one. Sensing something might be wrong with year one. Sensing something might be wrong with the data, and after determining there was no the data, and after determining there was no documentation of the steps taken to analyze year documentation of the steps taken to analyze year one data, the new staff person reanalyzed the first one data, the new staff person reanalyzed the first year’s data and found different results from what year’s data and found different results from what had already been reported. had already been reported.

7. Reporting7. Reporting

ScenarioScenario: The highly distributed executive : The highly distributed executive summary of program results that was released summary of program results that was released to the funder showed that 90% of the program to the funder showed that 90% of the program participants completed the program. The data participants completed the program. The data table in the released full report showed only 70% table in the released full report showed only 70% completed. The actual output from the statistical completed. The actual output from the statistical analysis showed that 70% of the participants analysis showed that 70% of the participants completed the program. The clerical staff completed the program. The clerical staff inadvertently typed a “9” instead of a “7” in the inadvertently typed a “9” instead of a “7” in the executive summary. The program director had to executive summary. The program director had to address the difference (admit to the error) with address the difference (admit to the error) with the funder.the funder.

8. Data storage and security8. Data storage and security

ScenarioScenario: Due to a hardware failure the : Due to a hardware failure the data files were not accessible. Since data data files were not accessible. Since data were not backed up on a regular basis, the were not backed up on a regular basis, the two year’s worth of program data two year’s worth of program data completed to date was lost, just before the completed to date was lost, just before the final analyses for the program were final analyses for the program were addressed. Final program performance addressed. Final program performance reports to the board of directors and reports to the board of directors and funders could not be produced. funders could not be produced.

Wrap Up Q&AWrap Up Q&A