AD-A270 507

DLA-93-P30054

DEFENSE CONTRACT MANAGEMENTCOMMAND DATA VALIDATION FILTER

June 1993 DTIC - T

r, ISI Q j ELECT-rES, D

FOR

mw DEPARTMENT OF DEFENSE

N1) DEFENSE LOGISTICS AGENCY

DEFENSE CONTRACT MANAGEMENT COMMANDCameron Station

4;b& Alexandria, VA 22304-6100

__ INSIGHT THROUGH ANALYSIS

DOR0 CORPORATE RESEARCH

DLA-93-P30054

DEFENSE CONTRACT MANAGEMENT

COMMAND DATA VALIDATION FILTER

June 1993

John S. McKinney

James Russell Edward J. Modic

CORPORATE RESEARCH TEAM OPERATIONS RESEARCH OFFICE

DEPARTMENT OF DEFENSE

DEFENSE LOGISTICS AGENCY

Executive Director (Plans & Policy Integration)Cameron Station

Alexandria, VA 22304-6100

DEFENSE LOGISTICS AGENCYHEADQUARTERS

CAMERON STATIONALEXANDRIA, VIRGINIA 22304-6100

CAI

FOREWORD

This Defense Logistics Agency Operations Research Office (DORO)report provides information regarding the development andapplication of a computer based data screening tool. DefenseContract Management Command (DCMC) will use this model initiallyto validate the monthly unit cost counts and later will begintesting a wide variety of management information data.

CHRISTINE L. GALLOExecutive DirectorPlans and Policy Integration

A08ess1on For

NTIS gRA&IDTIC TAB 0unanounced 0Justificatio

ByDistribut j•f/ _ ..Availability ro.

Special

DTIC QUALITY INSPECTED 8

TABLE OF CONTENTS

Section Title Page

FOREWORD iii

1 INTRODUCTION 1-1

2 METHODOLOGY 2-12.1 Statistical Process Control 2-1

2.2 Single Exponential Smoothing 2-1

2.3 Data Validation Model 2-22.4 Usage of Model 2-3

3 CONCLUSIONS 3-1

4 RECOMMENDATIONS 4-1

APPENDIX: USER'S GUIDE A-1

v

SECTION 1INTRODUCTION

The Defense Contract Management Command (DCMC) is becomingincreasingly dependent on accurate workload reporting due tothe adoption of unit cost based resourcing. DCMC managementhas identified approximately 150 data elements that will beused to monitor DCMC activity. Eighteen of these data elementsfeed the unit cost system. Others will be used to trackperformance. Inaccurate values for these data elements willresult in improper unit costs, resource levels, and performancemeasurements.

DCMC's data problems in the past have included missing data(usually not input on time), partial data reporting, anderroneous data input. Frequently erroneous input will involveone or more extra zeroes in a number. With the advent of theunit cost system, some new data elements are being reported.This introduces other possible errors, for example, usingdifferent units of count in different Secondary Level FieldActivities (SLFAs), or even within the same SLFA.

The recent sweeping DCMC organizational changes have impacteddata accuracy. The realignment of DCMC into five districts andthe consolidation of Military Service activities into DCMC aresome examples of these changes. Mechanization of ContractAdministration Services (MOCAS) data bases have been fragmentedfor each district during these transitions. This fragmentationhas made complete and accurate data collection difficult.Additionally, some interface problems still exist betweenformer Military Service activities and MOCAS.

While some of these problems have been resolved, attentionneeds to be focused on identifying and correcting data errors.One way to help increase data accuracy is to develop tools thatcan be used by DCMC personnel during data input and reporting.

i-i

SECTION 2

METHODOLOGY

2.1 STATISTICAL PROCESS CONTROL

Statistical Process Control (SPC) is one technique that is alogical choice for the problem of data validation. SPC iswidely used in industry to monitor and improve manufacturingprocesses. Control charts are used to plot a series ofmeasurements for an important characteristic of the output(e.g., hardness of a composite metal or width of a piece ofcloth). This chart has a center line (which is usually theaverage value) along with control limits at 1, 2, and 3standard deviations above and below the center line. Theprobability of getting output with only random variation whosemeasurement falls outside of three standard deviations is onlyabout 1 percent, based on a normal distribution of outputvalues. Certain patterns that appear in this plotted data canalso provide useful information. Principles of SPC are used inthis model to help determine a reasonable range of values totest data input. The same attributes of SPC that make it sucha powerful manufacturing tool can be adapted to also make it auseful tool for data validation.

While SPC is an effective tool, there are only certain types oferrors it will catch when validating data input. Errors inmagnitude will be effectively highlighted. For example, if theincorrect value 60 is input when 6,000 is the correct value,the incorrect value would be flagged. However, if theincorrect value 60 is input and the correct value is 53, thiserror would not necessarily be detected. Also, in cases wheretrends exist, traditional SPC by itself will not effectivelyaccommodate these trends. Any useful data validation modelmust be able to react and adjust to trends as the Departmentof Defense continues its downsizing and DCMC workload declines.

2.2 SINGLE EXPONENTIAL SMOOTHING

We increased the effectiveness of our model in handlingtrends by combining SPC with a forecasting technique calledsingle exponential smoothing (SES). SES is a widely usedforecasting technique that predicts a future value by focusingon the most recent actual values. Older values receive(exponentially) smaller weights. For many data sets, we wouldexpect that the latest values in the series would be betterpredictors for the next period than older values. SES willenhance our model by blending the effects of trend into ourrange of acceptable values.

2-1

The SES technique requires that a smoothing factor between zeroand one be selected, depending on the amount of smoothing thatis desired in the forecasts. This smoothing factor is calledalpha. Smoothing refers to the amount of variation in theforecasts from period to period. Alpha is simply a weight thatwe assign to place more or less emphasis on the latest datavalue. Using a small alpha (0.1) will cause forecasts to bevery smooth and not very sensitive to changes in the data. Alarge alpha (0.9) will give forecasts that are not very smooth.The optimum alpha level will lead to forecasts with the leastamount of error; this optimum level can vary over time as datacharacteristics change. The model selects the optimal alpha byminimizing the Mean Absolute Percent Error (MAPE) each time aforecast is made.

2.3 DATA VALIDATION MODEL

By combining SPC and SES techniques, we can create a range ofreasonable values for our target data. The SES forecast for aperiod (in our case, one month) is compared with the actualvalue for that period. The difference between the forecast andthe actual value is the error in the forecast. The average ofpast forecast errors serves as a mid-line for our pseudocontrol chart. The range around the mid-line is calculated bymultiplying the standard deviation of the forecast errors by1.96 and -1.96. This range captures approximately 95 percentof expected values, based on a normal distribution. When a newactual value is added to the model, the amount of error in theforecast is compared to the computed range of reasonableerrors. If the amount of error falls outside of this range,the model will flag the value. This warns the user that thisvalue is statistically unusual and requires review.

If the user finds that the value is incorrect and changes itin the model, the model will recalculate the amount of errorand create a new range for reasonable error values, the nexttime a forecast is made. If the actual data is zero, the modelestimates these as "missing" values, if values for adjacentmonths are greater then 0. Data that is considered "missing"remains flagged, with the value of 0 (in the data base), butfor calculation purposes only a value is substituted. If onlyone data point is "missing," the estimate used for forecastingwill be the average of the values before and after the"missing" value. If two consecutive data points are "missing",interpolation between the two next closest values is used. Forexample, if the values for January and February 1993 are"missing," and the value for December 1992 is 50 and the valuefor March 1993 is 62, the model would use 54 for January and 58for February.

2-2

2.4 USAGE OF MODEL

The model could be used at either the point of data inputand/or after District level data is compiled. The primaryconsideration for where to use the model was accountability forwhether or not changes to the data would be made when required.If the model was only used at the point of input, Districtlevel personnel might not be aware if values were verified orcorrected. If, however, the model was run at the Districtlevel, District personnel would be responsible for seeing thatthe required changes are made.

The accountability issue doesn't preclude using the model bothat the point of input and at the District level but otherproblems surfaced concerning using the model at the inputlevel. First, there is an effort underway to automate the UnitCost work counts where no manual input will be required. Thiswill not happen immediately for all work counts, but some maybe automated in the very near future. This would mean theseautomated counts could not be run through the model if it wasused at the point of input. The model could validate all workcounts, automated or not, if implemented after the Districtlevel data was compiled.

There are also differences among the Districts as to who inputscertain work counts. There was also some confusion as towhether all data for a work count for an SLFA was input at thesame location. This would require even more customization thanwould be required even if data was validated at the point ofinput. However, only five programs and data bases are requiredif data is validated at the District level.

2-3

SECTION 3

CONCLUSIONS

Accuracy in the reporting of unit cost and other managementinformation data is essential for DCMC. Inaccurate reportingwill lead to inappropriate workload and resource assessmentsand reduced efficiency.

Use of this model will increase reporting accuracy and database integrity. The model should be applied to the unit costdata at the District level after it has been input at the SLFAlevel . Validating the data at the point of input at the SLFAproved impractical. Many of the monthly counts will soon beextracted automatically from MOCAS and other data systems,which precludes validating the data at the SLFA. Additionally,there will be more control over this validation effort byhaving a small group responsible for investigating flaggedvalues and making corrections.

Personnel with all levels of data validation expertise willbenefit from using this statistical model to highlight certaintypes of possible errors for further review.

3-1

SECTION 4

RECOMMENDATIONS

The Data Validation Filter should be used for validating unitcost work counts by both DCMC headquarters and district levelpersonnel.

After testing and review, the model should be applied to keydata elements used to monitor DCMC activity.

Individuals should be designated (by position and name) at eachDistrict as well as Headquarters Defense Logistics Agency toaggressively investigate and change (if necessary) all flaggeddata values.

4-1

APPENDIX A

DCMC DATA VALIDATION FILTER (DDVF) USER'S GUIDE

A-i

SECTION 1OVERVIEW

The DCMC Data Validation Filter (DDVF) highlights possibleerrors in the Unit Cost work counts (there are currently 17 dataelements being collected). The DDVF allows for collecting thedata being checked (current month) with minimal effort on thepart of the user. This required data will be extracted monthlyfrom the Management Analysis Statistical System (MASS) and sentto the Districts along with the data used for the Unit Costreports. The user will only have to download the data from theDistributed Mini System (DMINS) in the usual manner. NOTE: thefile must be downloaded as a word processing (not binary) file.Each District will then run a stand alone, menu-driven programon their Personal Computer (PC) which will validate the data andhighlight data that should be corrected. Any changes to be madewill be made in another part of the model. A third menuselection will allow the user to display the 12 month history(if available) for any work count (one at a time) for anySecondary Level Field Activity (SLFA) (also one at a time).

Because it is expected that the Districts will have to ask theirSLFAs for any data corrections, the correction module wasdesigned separately from the validation portion of the model.This allows the user to complete validation, exit the program,and then re-enter the model to make corrections.

SECTION 2INSTALLING THE MODEL

The model runs on a PC with a minimum of 640 kilobytes ofmemory, a hard disk drive (not more than 2 megabytes of freememory is necessary), either a 3.5 or 5.25 floppy drive, andconnectivity to DMINS to allow file downloading.

The word <enter> will be used in this guide any time the user isrequired to press the enter key. Double quotes " " will be usedto highlight required keystrokes. The actual double quotemarks should not be typed.

To install the DDVF model:1. Turn the computer on.2. Put the DDVF PROGRAM DISK in the A: drive (Floppy).

(Use the 3.5 or 5.25 floppy disks depending on whetherthe A: drive is a 3.5 or 5.25 drive.)

3. At the C> prompt, create a directory for the model,by typing "md ddvf".

4. Still at the C> prompt, change to the DDVF directory,by typing "cd ddvf".

5. You must be at the C:\DDVF> prompt. Copy the DDVF modeland all necessary data bases to the DDVF directory,by typing "copy a:*.*".

A-3

6. To run the model, type the file name listed in Table 1for each District (don't type ".EXE"). For example, forthe Southern District, type "DCMDSDVF".

7. To validate new data, remember it has to be downloadedfirst. Download the designated file will be used foreach District. Make sure you download it to the c:\DDVFDirectory, and that the downloaded file is named exactlyas shown the last column of Table 1. Once the properlynamed file is in the DDVF Directory, start the model thesame as shown in step 6.

Table 1. District Data Filenames

District .EXE Filename Downloaded FilenameDCMDS DCMDSDVF.EXE ATLPSNS.ASCDCMDN DCMDNDVF.EXE BOSPSNS.ASCDCMDC DCMDCDVF.EXE CHIPSNS.ASCDCMDW DCMDWDVF.EXE LAPSNS.ASCDCMDM DCMDMDVF.EXE PHIPSNS.ASC

SECTION 3USING THE MODEL

Performing steps 6 and 7 from Section 2 above will get the userto the main menu. The main menu will have three functions tochoose from (four if you count Exit).

Selecting 1 allows the user to validate current work counts.Remember, you have to download the file to the DDVF Directoryand name it properly (See steps 6 and 7 in Section 2) beforeusing this menu item. The output of this function is a displayof all the work counts for the District. Those values outsidethe limits calculated by the model are highlighted (the screencolor is different) as possible errors.

Menu item 2 is the data correction function. Answering somescreen prompts will allow the user to change data for anyprevious month (which also includes the current month, verifiedin menu function 1). When prior data is changed, the flag forthat data is removed. The next time new data is validated, thechanged value is used to calculate the forecast and flags arereset. Therefore, it is possible that a value may be changed(when changed the flag will be removed), but the data may beflagged again when the next month's data is validated. Thisjust means that the changed value is still outside the model'slimits. Remember, the model highlights possible errors.

Menu function 3 displays and prints historical monthly data bySLFA. The data displayed is the current (most recentlyvalidated month) and the previous 6 months, for all the workcounts. As on the validation screen, possible errors arehighlighted

A-4

To print any of the display screens, simply press the "PrintScreen" key. Since the highlighted values cannot be printed ina different color, possible errors are highlighted during screenprinting with an "*" after the highlighted value (this "*" doesnot show up during screen display).

A-5

REPORT DOCUMENTATION PAGE 0 J11. AGENCY USE ONLY (leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED

June 1993 Final4. TITLE AND SUBTITLE S. FUNDING NUMBERS

Defense Contract Management Command Data ValidationFilter

6. AUTHOR(S)

John S. McKinneyEdward J. Modic

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION

HQ Defense Logistics Agency REPORT NUMBER

Operations Research and Economic Analysis Office (DORO) DLA-93-P30054c/o Defense General Supply CenterRichmond, VA 23297-5000

"9. SPONSORING ,' MONITORING AGENCY NAME(S) AND ADORESS(ES) 10. SPONSORING" MONITORINGHQ Defense Logistics Agency AGENCY REPORT NUMBER

Corporate Research Team (CAILR)Cameron StationAlexandria, VA 22304-6100

11. SUPPLEMENTARY NOTES

12a. DISTRIBUTION /AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE

Unlimited distribution; public release

13. ABSTRACT (Maximum 200 words)

This report documents the development and application of a computer based datavalidation tool for use by the Defense Contract Management Command (DCMC).The model is based on Statistical Process Control (SPC) principles combinedwith Single Exponential Smoothing (SES) forecasting. Monthly data values thatare statistically unusual compared to historical values are flagged for reviewand possible correction. This kind of model is necessary for the effectiveimplementation of the unit cost based resourcing system. Inaccurate unit costsystem data will result in inappropriate workload and resource assessments.This model should eventually be used to validate a much wider array of keymanagement data at the DCMC, Secondary Level Field Activity (SLFA) andDistrict levels.

14. SUBJECT TERMS 15. NUMBER OF PAGES

23

unit cost, data validation model, DCMC 16. PRICE CODE

17. SECURITY CLASSIFICATION 18. SECURITY CLASSIFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF ABSTRACTOF REPORT OF THIS PAGE OF ABSTRACT

UNCLASSIFIED UNCLASSIFIED UNCLASSIFIEDNSN 7540-01-280-5500 Standard Form 298 (Rey 2-89)

P,"cr,brd by ANS] "td :39.-0