Page 1

Presented by Lee GiesbrechtOffice of Audits & EvaluationsInspector GeneralDepartment of Veterans Affairs

Presented by Lee GiesbrechtOffice of Audits & EvaluationsInspector GeneralDepartment of Veterans Affairs

StatisticalSamplingStatisticalSampling

2

What is Sampling?

• Check a small part of something to get an ideaabout the whole thing.

• For example…– Sip a spoon of soup to check the temperature of

the whole pot.

– Test a small sample of blood to check yourcholesterol levels.

– Other examples?

Page 2

3

Why Select a Sample?

Need to know something about a largepopulation, but its impractical to collect dataon every member of the population because:– The needed information about the population is

not on one central computer system or easilycompiled.

– It is too expensive or time-consuming to collectdata for all population members (travel, timeconstraints, personnel constraints, limitedbudget).

Two Kinds of Sampling

• Judgment or Convenience Sampling– Select sample units based on knowledge and

experience.– May result in representative data, but no way to prove

it scientifically.– Results cannot be generalized to the population.

• Statistical or Random Sampling– Introduce a component of random chance into the

selection process.– Can prove that data are representative (unbiased)

using statistical methods.

Page 3

Steps to Sampling

• Put audit goals into quantifiable terms –specifically what estimates of the populationare needed?

• Identify population of interest.

• Identify sampling frame(s) – what data areavailable for each population member?

• Design sample based on data needs andpractical concerns.

• Select sample – use random numbers.

6

Quantifiable Audit Goals

• Determine what estimates of the populationare needed – be as specific as possible.

• Do the analysis first – this will help to pinpointthe needed estimates.– Create “table shells” (empty tables) that will be

filled with numbers after data analysis is done.

– Identify subgroup analyses.

– Compare scenarios of possible outcomes tofurther help determine analytical needs.

Page 4

Statistical (Random) Sampling

• Sample selection is done using a random (notdeterministic) process.

• Each sample is one of a large number ofpossible samples.

• Results (i.e., means) from all possible samplestogether form a sampling distribution.

• The sampling distribution is centered on thetrue population mean.

8

Sampling Distribution

Page 5

9

What is Sampling Error?

• A.k.a. standard error, precision.• Difference between the projection and the

true (unknown) value in the population.• This difference is due to the randomness of

the sample.• Probability theory allows us to measure the

amount of sampling error for each projectionfrom our sample data.

• Use this to create a confidence interval (a.k.a.margin of error).

10

Computing Sampling Error

)1(

2

N

XXS i

Page 6

Sample Size

• Like most things, it is a balance of quality(precision), timeliness, and cost.

• Larger samples reduce sampling error.

• Decisions about the needed sample size take thesample design into account – stratification,clustering, and other complexities impact theprecision and, therefore, the needed size.

• In general, reducing sampling error by ½ requiresa four fold increase in sample size.

12

Sample Size and Sampling Error

ErrorRate

SampleSize

Plus orMinus

15% 50 8%

15% 200 4%

Page 7

13

Confidence Interval (Margin of Error)

Simple Random Sample

• Basis for all other sampling designs.• All members of the population have an equal

chance of being selected into the sample.• Easiest type of sample to analyze due to

simplicity of computing margins of error.• Often not practical to use in situations where

data are spread across geography.• Doesn’t take advantage of information you have

about the population to improveprecision/reduce the margins of error.

Page 8

15

A Little About “Randomness”

• Statistical sampling works when a sample isselected through a non-deterministic process.

• True random numbers cannot be generated bya computer – computers are completelydeterministic machines!

• True randomness is an “analog” process.

• Random numbers from a computer areactually pseudo-random – generated from aformula that can have a predictable outcome.

16

Simple Random Sample Example

• Use Random.Org to create a randomsequence of numbers.

• Copy random sequence column into samplingframe spreadsheet.

• Sort the list of records (sampling frame) by therandom sequence.

• Sample can be as large as needed, as long asrecords are used in order.

Page 9

17

18

Page 10

19

Random

Order

Original

OrderRO Number RO Name

7 1 301 Boston

10 2 304 Providence

9 3 306 New York

1 4 307 Buffalo

6 5 308 Hartford

5 6 309 Newark

3 7 310 Philadelphia

11 8 311 Pittsburgh

4 9 313 Baltimore

12 10 314 Roanoke

2 11 315 Huntington

8 12 316 Atlanta

20

Random

Order

Original

OrderRO Number RO Name

1 4 307 Buffalo

2 11 315 Huntington

3 7 310 Philadelphia

4 9 313 Baltimore

5 6 309 Newark

6 5 308 Hartford

7 1 301 Boston

8 12 316 Atlanta

9 3 306 New York

10 2 304 Providence

11 8 311 Pittsburgh

12 10 314 Roanoke

Page 11

Stratification

• Uses information from the sampling frame toforce the sample to be more representative ofthe population.

• Reduces sampling error because it ensuresthat the sample is more like the population.

• Form strata based on:– Amount/size of transaction

– Dates of transaction / type of transaction

– Location of sample units

Design Strata to…

• Ensure that the sample contains the sameproportion of population subgroups as exist inthe population.

• Ensure sample is spread across geography.

• Minimize the variance of a comparisonbetween groups.

Page 12

23

Stratification Example

• Stratified by percent incomplete C&P Exams.– Stratum A – highest percent

– Stratum B – middle percent

– Stratum C – lowest percent

• Ensures stations with different percentsincomplete C&P exams are in sample.

• Selected (almost) equal sized sample for eachstratum to minimize the variance ofcomparisons between strata.

24

Systematic Sampling

• Works when you have a complete list ofpopulation members.

• After determining sample size…

– Divide population by sample size (= interval).

– Select random number between 1 and interval.

– Select that record and every one after that byadding the interval.

• Can also sort frame for implicit stratification.

Page 13

25

Advantages/Disadvantages

• Simple Random Sampling– Advantage – Can quit work early – as long as sample

units are completed in order of selection.– Disadvantage – May be difficult to implement in the

field.

• Systematic Sampling– Advantages:

• Easier to select sample.• “Implicit” stratification by sorting list.

– Disadvantage – Can’t scale – must complete all sampleunits.

Clustered, Multi-Stage Samples

• Used when data must be collected at severallocations or there is no national frame.

• For example, we select a two-stage samplewith the first stage being a sample of officesand the second stage being claims folders.

• Sample is geographically “clustered” withineach sampled office.

Page 14

Multi-Stage Sample Example

• First stage sample of VISNs.

• Second stage sample of VAMCs.

• Third stage sample of appointments withineach sampled VAMC.

Effects of Clustering

• Reduces travel costs since fewer sites must bevisited.

• Reduces time required for data collection dueto less travel time.

• Generally reduces precision due to intra-cluster correlation.

• Two sources of sampling error:– Within cluster variation

– Between cluster variation

28

Page 15

Probability Proportional to Size

• Each unit sampled in proportion to its “size”.

• Examples of size measures:– Number of appointments

– Amount of payment

– Number of claims

• Ensures that “larger” units are more likely tobe sampled.

• Reduces sampling error for estimates relatedto the size measure.

Discovery Sampling

• Used to detect presence of rare events such asfraud and avoidance of internal controls.

• Select sample to have a specified probabilityof including at least one rare event based onan assumption of the prevalence of that eventin the population.

• If the rare item is found, can stop datacollection and report that the prevalence is atleast as high as that specified.

Page 16

Analysis of Sample Data

• Weighting

• Post stratification

• Compute estimates and sampling errors

• Interpret analytical findings

• Statistical significance testing

• Present analytical findings in audit report

What is a Sampling Weight?

• Sampling weights indicate how manypopulation members are represented by eachsample unit.

• In the simplest case, each sample unit has thesame weight and represents the same numberof population units (simple random sample).

• However, weights may not be the same for allsample units due to a more complex design.

Page 17

How are Weights Computed?

• Weights are the inverse of the probability of selection.

• For example:– Simple random sample of 100 units.

– Population of 5,000.

– Probability of selection is 100 in 5,000 = 1/50.

• Weight will equal the inverse, 50/1 or 50.

• Each sample unit represents 50 units in the population.

• Complex samples with unequal weights make thiscalculation more complicated.

34

Weighted vs. Unweighted Data

• With unweighted data, each sample unit iscounted equally.

• Unweighted data represent only those in thesample who provided data.

• With weighted data, each case is countedrelative to its representation in thepopulation.

• Weights allow analyses that represent thetarget population.

Page 18

Why are Weights Important?

• Weights allow us to calculate populationestimates.

• To estimate a population total, sum theweights.

• When weights for all sample units are thesame, population percentages equal samplepercentages.

• If weights are not the same for all sampleunits, must compute weighted percentages.

Why Unequal Weights?

• Stratification – i.e., each stratum has differentpopulation size but the same sample size.

• PPS sampling – each unit has a probability ofselection proportionate to its measure of size.

• Post-stratification – weights are adjusted aftersampling to conform to known populationtotals.

Page 19

Unequal Weights Example

StratumPopulation

SizeSample

SizeProbability of

SelectionSamplingWeight

1 5,000 100 1/50 50

2 8,000 100 1/80 80

3 2,000 100 1/20 20

4 12,000 100 1/120 120

Error Rate Example

Unweighted Error Rate: 79 ÷ 200 = 40%

Weighted Error Rate: 4,080 ÷ 14,000 = 29%

StratumPopulation

SizeSample

SizeSamplingWeight

ErrorsFound

WeightedErrors

1 2,000 100 20 54 1,080

2 12,000 100 120 25 3,000

Total 14,000 200 79 4,080

Page 20

Post Stratification

• Adjust the sample weights so that sampleestimates of known population values equalthose values.

• The population values used must be of higheraccuracy and precision than the sampleestimates.

• Improves accuracy and reduces bias of sampleestimates.

Prepare Data for Analysis

• Unless data were collected using a simplerandom sample, a micro data file (database)must be prepared.

• Basically, a spreadsheet where the rowscontain records and the columns containfields.

Page 21

What Estimates Are Needed?

• Error Rates– Overall – for the total sample.

– By strata or other characteristics.

– This is a frequency count of an attribute.

– Can get decent precision with a small sample.

• Impacts of Errors– Money that could have been saved.

– Time that could have been saved.

– This is a sum (mean, median) of a variable.

– Need larger sample to get decent precision.

41

Compute Sampling Errors

• If data are from a simple random sample,many programs will calculate this correctly.

• If data are from a complex sample design(stratified, clustered, unequal probabilities ofselection) must use other methods.– Taylor Series Linearization/Approximation –

approximates a linear function for a series of non-linear equations.

– Replication – uses the sample data to simulate theact of selecting multiple samples.

Page 22

43

Complex Sample Software

• SUDAAN – replication or Taylor SeriesLinearization.

• Stata – replication or Taylor Series Linearization.

• SAS – replication, ultimate clusters, directestimation (labor intensive programming).

• VPLX – replication (freeware created by Dr.Robert Fay at the Census Bureau).

• WesVar – replication.

• Others

What is Significance Testing?

• Demonstrate whether the difference betweentwo (or more) sample estimates is statisticallysignificant or if the difference could have beencaused by chance alone (sampling error).

• Takes the sampling error of the estimates intoaccount when comparing them.

• We sometimes compare estimates withanother number, for example, a numberreported by the audit client.

Page 23

How to Test for Significance?

• First compute the sampling error for eachestimate to be compared.

• Then compute the difference between theestimates.

• Then compute the sampling error of thedifference.

• If the confidence interval of the differencebetween the two estimates contains zero, thedifference is not statistically significant.

Significance Testing Example

Page 24

• All estimates mentioned in the report are listed in thesampling appendix with associated margins of error.

• The margins of error impact how estimates are stated.

• For example, if an estimate from the sample is1,624,981 transactions are in error with a margin oferror of 209,378, the report may state:– “About 1.6 million transactions are in error.”

– “About 1.6 million (± 200,000) transactions are in error.”

– “Between 1.4 million and 1.8 million transactions are inerror.”

Reporting Audit Results

Statistical Appendix

• Includes a short description of sampling andestimation procedures.

• Includes tables showing all estimates in thereport along with their margins of error.

• The confidence interval is stated in thesampling appendix – OIG policy is to use a 90percent confidence interval to compute themargins of error for sample estimatesreported in audits.

Page 25

Appendix Table Format

ErrorRates

EstimateMargin of Error Based

on 90% ConfidenceInterval

90% Confidence Interval

Lower Limit Upper Limit

Type 1 20% 8% 12% 28%

Type 2 25% 7% 18% 32%

Type 3 30% 6% 24% 36%

Note: the margins of error and confidence intervalsare indicators of the precision of the estimates. If alarge number of samples were selected and estimateswere computed from each one, 90 percent of thoseestimates would fall within the confidence interval.