Medicare and Medicaid Audits Using Statistical...

Medicare and Medicaid Audits Using

Statistical Sampling and Extrapolation:

Challenging Methods and Results

Today’s faculty features:

1pm Eastern | 12pm Central | 11am Mountain | 10am Pacific

The audio portion of the conference may be accessed via the telephone or by using your computer's

speakers. Please refer to the instructions emailed to registrants for additional information. If you

have any questions, please contact Customer Service at 1-800-926-7926 ext. 1.

THURSDAY, JUNE 14, 2018

Presenting a live 90-minute webinar with interactive Q&A

Anna M. Grizzle, Member, Bass Berry & Sims, Nashville, Tenn.

Dr. Patricia L. Maykuth, Ph.D, President, Research Design Associates, Decatur, Ga.

Tips for Optimal Quality

Sound Quality

If you are listening via your computer speakers, please note that the quality

of your sound will vary depending on the speed and quality of your internet

connection.

If the sound quality is not satisfactory, you may listen via the phone: dial

1-866-927-5568 and enter your PIN when prompted. Otherwise, please

send us a chat or e-mail sound@straffordpub.com immediately so we can address

the problem.

If you dialed in and have any difficulties during the call, press *0 for assistance.

Viewing Quality

To maximize your screen, press the F11 key on your keyboard. To exit full screen,

press the F11 key again.

FOR LIVE EVENT ONLY

Continuing Education Credits

In order for us to process your continuing education credit, you must confirm your

participation in this webinar by completing and submitting the Attendance

Affirmation/Evaluation after the webinar.

A link to the Attendance Affirmation/Evaluation will be in the thank you email

that you will receive immediately following the program.

For additional information about continuing education, call us at 1-800-926-7926

ext. 2.

FOR LIVE EVENT ONLY

Program Materials

If you have not printed the conference materials for this program, please

complete the following steps:

• Click on the ^ symbol next to “Conference Materials” in the middle of the left-

hand column on your screen.

• Click on the tab labeled “Handouts” that appears, and there you will see a

PDF of the slides for today's program.

• Double click on the PDF and a separate page will open.

• Print the slides by clicking on the printer icon.

FOR LIVE EVENT ONLY

Medicare and Medicaid Audits Using Statistical Sampling and Extrapolation

Anna Grizzle - Bass, Berry & Sims, PLCPat Maykuth, Ph.D. - Research Design Associates, Inc.

Use of Statistical Sampling for Overpayment Estimation

• Medicare or Medicaid audit

– MAC audits following TPE reviews

– ZPIC or UPIC audits

– OIG audits

– Medicaid agency audits

– MIC audits

• OIG self-disclosure protocol

• Internal compliance audit

• Calculation of damages in FCA case?

6

Legal Basis for Statistical Sampling for Overpayment Estimation

“The use of statistical sampling to project anoverpayment. . . does not deny a provider orsupplier due process. Neither the statute norregulations require that a case-by-case reviewbe conducted in order to determine that aprovider or supplier has been overpaid and todetermine the amount of overpayment.”

HCFA Ruling 86-1

7

Legal Basis for Statistical Sampling for Overpayment Estimation

Statistical sampling does not violate dueprocess “so long as extrapolation is madefrom a representative sample and isstatistically significant.”

Chaves County Home Health Service, Inc. v.Sullivan, 931 F.2d 914 (D.C. Cir. 1991), cert.denied, 402 U.S. 1091 (1992).

8

Legal Basis for Medicare Statistical Sampling and Extrapolation

A Medicare contractor may not use extrapolation to determine overpayment amounts . . . unless . . .

– There is a sustained or high level of payment error; or

– Documented educational intervention has failed to correct the payment error

42 U.S.C. §1395ddd(f)(3)

9

Legal Basis for Medicare Statistical Sampling and Extrapolation

• The PIM provides basic concepts rather than a checklist to complete a valid statistical sampling.

– When applied correctly, the PIM’s concepts can lead to a proper methodology to use as a basis for extrapolation.

• The PIM’s concepts often are not applied correctly in developing the statistical sampling methodology used as the basis of extrapolation in Medicare audits.

Source: Chapter 8 – Benefit Integrity; Medicare Program Integrity Manual; available at:

http://www.cms.gov/manuals/downloads/pim83c08.pdf10

Legal Basis for Medicaid Statistical Sampling and Extrapolation

• Dictated by state law

• If no explicit authority, look to due process requirements

11

Performance of Statistical Sampling and Extrapolated Overpayment

• Major Steps

– Selecting the provider or supplier• Prior history of overpayment error

• TPE Review by MACs

– Selecting the period to be reviewed

– Defining the universe, the sampling unit, and the sampling frame• Define the provider, issue to be reviewed, time

period, and methodology for measuring overpayment

Source: Chapter 8 – Benefit Integrity; Medicare Program Integrity Manual; available at:

http://www.cms.gov/manuals/downloads/pim83c08.pdf

12

Performance of Statistical Sampling and Extrapolated Overpayment

• Major Steps (cont.)

– Designing the sampling plan and selecting the sample

– Reviewing each of the sampling units and determining if there was an overpayment or under payment

– Estimating the overpayment

Source: Chapter 8 – Benefit Integrity; Medicare Program Integrity Manual; available at:

http://www.cms.gov/manuals/downloads/pim83c08.pdf

13

Documentation Requirements

• The following items must be documented:

– Universe

– Sampling frame (sorted)

– Random sampling process (seed, program, inputs and printout)

– Sample size determination calculations

– Extrapolation formulae, inputs and printouts

– Extrapolation recalculation where appropriate

14

Valid Methodologies

• MPIM specifically lists:

– Simple random

– Stratified random

– Cluster

• MPIM specifically requires proper execution of chosen methodology

15

Valid Statistics

• Statistically Valid Random Sample (SVRS)

• Probability Sample

• Use correct formulae

• Follow statistical requirements of chosen statistics

• Point Estimates and Confidence Intervals are valid statistics – if properly executed

• Minimum Sum Method and Penny sampling not been validated in the statistical literature

16

Probability Statistics • To know the number of samples of the chosen size

that can be created from the frame

• Known likelihood of selection of each sampling unit

• Proper randomization

• Proper execution of sample methodology

• Use correct formulae

• Accurate measurement of the variable of interest (overpayment)

Source: Chapter 8 –§ 8.4.2 Benefit Integrity; Medicare Program Integrity Manual;

available at: http://www.cms.gov/manuals/downloads/pim83c08.pdf17

Frame

x

Sample

(1, 2, 3, 4, 5) (2, 3, 4, 5, 6) (1, 11, 21, 31, 41) (1, 12, 23, 34, 45) (8, 23, 46, 73, 90)

(1, 2, 3, 4, X)

Outside of frame

18

Probability Sample

1. To be able to identify the number of samples of a given size that can possibly be selected from a frame of a frame of a given size claims. How many samples of 5 can be selected from a frame of 100?

2. In a simple sample each claim must have a known and equal probability of selection. Strata take simple samples from strata frames.

19

Probability Yardstick• Single sample chosen for the audit is only one sample (of the chosen

size) out of a large number of possible samples

• Sampling distribution of all possible means provides mathematical model of what is likely to occur if all possible samples analyzed

• If a large number of samples were drawn from the frame:

– A mean can be calculated for each sample

– Each sample mean would not be exactly the same value as others

– Means would be different from one another but would cluster around the frame’s central value (or mean)

– Differences in the means of different samples are basis of the error that occurs inferring from a sample to the frame rather than measuring all of the claims in the frame

– If repeated random samples (moving toward infinity) were made, means would be expected to fall into a normal distribution

20

Possible Samples That Can Be Randomly Drawn From the Frame

21

What’s the Big Deal About the Normal Distribution?

• In probability theory, the normal (or bell-shaped) distribution is continuous probability distribution (a function that tells the probability of a number falling between any two real numbers). The normal distribution is symmetric around the mean. The mean, median and mode are the same number.

• The normal distribution is immensely useful because of the Central Limit Theorem which states that the mean of many random variables independently drawn from the same distribution is distributed approximately normally, irrespective of the form of the original distribution. That is, the overpayment means will be randomly distributed if the sample is large ... moving toward infinity.

22

“Normal” Distribution

23

Confidence Levels on Normal Distribution

Lower Confidence Limit

Point Estimate

Upper Confidence Limit

24

25

Mode

Mean

Median

0 12,072.62566.09 1,274.35

Where is the Confidence Level?One-sided or Two

Illustration not based on actual data

26

Distribution of Frame

Paid Dollars

27

Non Normal Mean and Standard Deviation

Sample mean +/- 1 standard deviation

Mean = 430.92 + sd = 525.51 = 956.43

430.92 - 25.51 = -94.18

28

Confidence Levels Non Normal Data

Point Estimate +/- ½ Confidence Interval

Confidence Interval = 16,778 + 16,778 = 33,556

Lower 27,147 + 16,778 = $43, 925

Upper 27,147 - 16,778 = $ 10,368

Lower confidence Level ???Unknowable

29

Properly Executed Random Sample

• Prior known error rate or probe

• Appropriate for use with the audit data (dependent or independent)

• Of sufficient size to support the statistic used

• Use a replicatable random process

• Tested for randomness

• Is representative (without mathematical bias) and tested

30

31

Sample Size Determination Based on Chosen Precision and

Confidence

32

RAT-STATS Results

80% 90% 95% 99%

1% 75 77 78 79

2% 63 68 71 75

5% 29* 39 46 56

10% 10* 15* 20* 29*

15% 5* 8* 10* 16*

Confidence Level

Precision

Level

* Caution sample sizes less than 30

33

Representative Samples

• A small yet complete and accurate picture of the data in the frame.

• A subset of a statistical universe that accurately reflects the numerical membership of the entire universe and its distribution.

• A representative sample is an unbiased indication of what the frame is like.

• Representativeness is tested mathematically. When a sample is not representative, the result is known as a sampling error.

34

Representative Sample

Frame

sample

35

Sobering How Poor a Sample Can Be

36

RAT-STATS Overpayment Estimation

• Formulae:

• Confidence Level:

37

Precision vs Accuracy

• Precision measures how often the measurement tool produces a similar result every time it is used.

• Accuracy measures how close the sample value comes to the true value. How close is the sample overpayment mean to the true mean of overpayments in the frame.

• Sample results can be accurate but not precise or precise but not accurate

• Extrapolations require that the results meet a predetermined level of both.

38

Precision Calculation

Point Estimate +/- Precision amount = Confidence Level

Precision amount/Point estimate = Precision %

It is absurd to say the lower confidence level benefits the provider. The accuracy of the estimate is only the top and bottom of the confidence level around the point estimate. The lower confidence level is as close to accurate overpayment that can be obtained.

39

Sample of 5

If:

• A random sample of 5 selects the top most expensive claims (5540, 5550, 5555, 5560, 5570) mean = 5555

• Frame of 100 ranges from 500 to 5570 has a mean = 2827• The precision will very small and the confidence levels

will be very tight but the point estimate would be nearly twice as large as the frame mean

• The precision would be good but the accuracy would not• Point estimates can exceed the amount actually paid• It is a bad estimate.

40

41

Sample of 5

If:• A random sample of 5 selects the top 2 claims and bottom 3

(500, 525 540, 5560, 5570) mean = 2539• Frame of 100 ranges from 500 to 5570 has a mean = 2827• The precision will very large and the confidence levels will be

very large but the point estimate would be close to the frame mean

• The precision would be unacceptable but the accuracy would be OK

• The lower confidence level can be a negative number indicating an underpayment

• It is a bad estimate

42

Poor Audit Design & Execution Produce Only “Invalid Results”

Statistics in the hands of an inept auditor are like a

lamppost to a drunk: they are used more for support than

illumination.

43

Defending Against Extrapolation Results

• Medicare Appeals Process

– Redetermination

– Reconsideration

– Administrative Law Judge Hearing

– Medicare Appeals Council

– Federal District Court

• Medicaid Appeals Process

– Appeal rights under state law

44

Defending Against Extrapolation Results

• No administrative or judicial review of determination of high level of payment error BUT determination must be made

• Failure to follow one or more requirements in Benefit Integrity Manual does not necessarily affect validity

• Not sufficient to argue better or more precise methods are available

45

Defending Against Extrapolation Results

• Can challenge validity of sampling methodology based on “the actual statistical validity of the sample as drawn and conducted”

• Test: Was the sample statistically valid?

• Contractor has burden of establishing sample was in fact random and statistically valid

46

Defending AgainstExtrapolation Results

• Procedural Challenges– Did the contractor provide work papers

allowing for review and replication of sampling results for every stage of the process?

– Did the contractor follow the guidelines?• Medicare: MPIM• Medicaid: State requirements

– Were allowed claims included in overpayment sample calculation?

– Were calculations performed correctly in the audit and at each level of appeal?

47

Defending Against Extrapolation Results

• Substantive Challenges

– Need a statistician to make your case.

• Where can you find one?

– “One size does NOT fit all.”

– It is not your job to explain how it should be done.

48

Defending Against Extrapolation Results

• Examples of Substantive Challenges

– Is the sample representative?

– Is the sample statistically significant?

• Is the sample size reliable?

• Is the sample within the required precision and confidence levels?

49

Hold Auditors Accountable• Did they know what should be done statistically?

– Requires consultation with statistical expert

• Did they do what is necessary to create and audit a probability sample (not just say they did)?

• Have to test the data to be sure

• Did they accurately follow chosen methodology key assumptions (accurately execute the methodology, use proper randomization, use the correct formulae and accurately measure variables of interest)?

• All of these criteria must be met for extrapolation

• Did they select a random sample of sufficient size made up of independent observations that are normally distributed, randomly selected and representative?

50

Common Substantive Issues

• Sample size not associated with established precision or confidence levels

• Incorrect use of formulas

• Use of wrong formulas - choose wrong method

• Use of inapplicable methodology – simple, stratified, cluster, multi-stage

• Non-representative sample

• Fail to meet key assumptions of statistics – math basis of statistics

• Exclusion of zero paid claims

• Accuracy outside of recommended range – too little precision

• Reporting incorrect precision and/or confidence levels51

Defending Against Extrapolation Results

• Obtain all documentation related to sampling calculations

– Consider provider’s prior audit history

• Know appeal timelines and requirements for each level

– No new evidence after reconsideration absent good cause

– Request for ALJ Hearing – 42 C.F.R. § 405.1014(a)(3)

• Include information on each sample claim to be appealed

• File request within 60 calendar days after the party receives the last reconsideration for the sample claims

• Assert the reasons for disagreement with how the statistical sample and/or extrapolation was conducted in the request for hearing

52

Defending Against Extrapolation Results

• Understand reasons for denial at each level

• Present reasons in written protest or position paper

• Prepare for oral testimony at ALJ hearing

53

Questions?

54

Faculty

Anna M. Grizzle, Esq.MemberBass, Berry & Sims PLC150 Third Avenue SouthSuite 2800Nashville, TN 37201agrizzle@bassberry.com(615) 742 7732

Pat Maykuth, Ph.D.PresidentResearch Design Associates, Inc.

721 E Ponce de LeonDecatur, GA 30030www.researchdesignassociates.compm@researchdesignassociates.com(404) 373 4637

55