Slide 1

Incentives in Surveys with Incentives in Surveys with FarmersFarmers

Third International Conference on Establishment SurveysMontreal, Quebec, Canada

June 19, 2007

Slide

Kathy Ott and Dan Becklerpresented by Norman Bennett

USDA, National Agricultural Statistics Service

Background Information - NASS

The National Agricultural Statistics Service (NASS) surveys Agricultural Operations (farmers/ranchers)

Collects information from the US agricultural sector on acreage, production, yield, economics, labor, etc.

Conducts over 400 surveys per year

Technically, these are establishment surveys, but have some similarities to household surveys

Slide 2

Background Information - ARMS

Agricultural Resource Management Survey (ARMS) is a long data collection, typically conducted by face-to-face interview

Asks detailed information on what is produced, economic items (assets, debt, income, expenses), and operator characteristics

Average interview time = 90 minutes

Typical national response rate = 60 to 65%

Very complex survey design with multiple versions. For this project, just two questionnaire versions used

Slide 3

Incentive Use

Goals for ARMS:• Raise overall response rate• Contain data collection costs• “Shift” to mail completion

Two Incentive Studies completed at NASS; both on the ARMS.

• 2004 Mail-out/Mail-back Study• 2005 Face to Face Interview Study

Slide 4

2004 Mail-out/Mail-back Study

2004 Reference Year, conducted early 2005

“Core” Questionnaire Version

Administered in 15 states, sample: ≈16,000

Mail-out/Mail-back data collection with face-to-face interview follow-up for all nonrespondents

Pre-Survey Letter First Questionnaire Mailing Postcard Reminder/Thank You Second QuestionnaireFace-to-face Interview follow-up on all mail nonrespondents

Slide 6

2004 Mail-out/Mail-back Study

Used a “monetary” incentive in the form of a $20 ATM card

Administered by: 1st class and priority mail Prepaid and promised

Treatment Groups receiving incentives included cover letters that:

Explained the incentive as a “Thank You” Described uniqueness of ARMS Justified incentive by overall cost-savings to government

Slide 7

ATM Cards

Cards loaded with $20 plus $4 for transaction fees

ATM Cards came pre-activated

ATM Card was affixed to an instruction sheet

Slide 8

Treatment Group

1st Questionnaire Mailing

2nd Questionnaire Mailing

1 (Control) 1st Class, no incentive 1st Class, no incentive

2 (1st, Prepaid) 1st Class, $20 ATM 1st Class, no incentive

3 (Priority) Priority, no incentive Priority, no incentive

4 (Priority, Prepaid) Priority, $20 ATM Priority, no incentive

5 (1st, Promised) 1st Class, promise $20 ATM

1st Class, promise $20 ATM

All groups also received a pre-survey letter, a post-card reminder, and face-to-face follow-up

2004 Treatment Groups

Slide 9

2004 Response Rate Results

Treatment Group N Mail ReturnsFace-to-Face Completes

Overall Completes

1 (Control) 1,948 30.1% 33.3% 63.4%

2 (1st, Prepaid) 1,941 40.8% 29.6% 70.4%

3 (Priority) 1,935 32.9% 31.8% 64.7%

4 (Priority, Prepaid) 1,952 43.9% 28.5% 72.4%

5 (1st, Promised) 1,946 37.2% 31.2% 68.4%

1,3

1,3,5

1,3

1,3

1,3

1

Red numbers indicate Treatment Groups to which this percent was significantly greater (at α = 0.05 from t-tests with Bonferroni adjustment).

Slide 10

2004 ATM Card Usage

Treatment GroupMail

RespondentsOverall

RespondentsOverall

NonrespondentsOverall Card Recipients

2 (1st, Prepaid) 60.4% 41.1% 4.7% 30.3%

4 (Priority, Prepaid) 58.9% 40.8% 5.4% 31.0%

5 (1st, Promised) 71.4% 61.4% 3.1% 61.3%

These people initially returned questionnaires which were later deemed inadequately completed.

Slide 11

2004 Overall Costs

Treatment Group

Postage & NPC Costs

All ATM Costs

Face- to- Face Costs

TotalAverage

per Sample

Average per

Complete

1 (Control) $11,520 -0- $176,291 $187,811 $96.41 $152.07

2 (1st, Prepaid)

$13,128 $14,845 $154,154 $182,127 $93.83 $133.33

3 (Priority) $21,722 -0- $170,951 $192,673 $99.57 $154.02

4 (Priority, Prepaid)

$23,250 $15,342 $148,356 $186,948 $95.77 $132.21

5 (1st, Promised)

$13,939 $20,486 $160,475 $194,900 $100.15 $146.32

Face-to-face follow-up costs.

Includes all ATM/POS withdrawals, all fees, and administrative costs.

Includes postage, printing, and NPC administrative fees.

Slide 12

Summary of 2004 Mail-out/Mail-back Study

Offering $20 ATM card incentives for the ARMS Core improved both mail and overall response rates

The incentives reduced overall costs, the average cost per sample and the average cost per complete.

The prepaid incentives performed better than the promised.

Priority mail further increased response rates (albeit non-significantly) when combined with the incentive but alone was ineffective.

Slide 13

What to do next?

Implemented use of prepaid incentives for entire Core version mail-out/mail-back sample (approximately 16,000 records)

For 2005, extended incentive research for face-to-face interview component of the data collection using the Cost and Returns Report (long) version of the ARMS questionnaire

Slide 14

Slide 15

2005 Face-to-Face Incentive Study

“Long” (32 pages) questionnaire version only

All 48 states included

Face-to-face interview only, no mail data collection

Presurvey letter was mailed with the incentive, followed by face-to-face interview data collection

Stimuli:Prepaid Monetary Incentive ($20 ATM card)“Prepaid” Non-monetary Incentive (Wall clock)Promised Individual Farm Analysis (IFA)

Treatment Group

First Contact Interview Contact

Pre-survey letter, plus: Face-to-face interview, plus:

1 (Control) No incentive No Incentive

2 (ATM Card) $20 ATM Card No Incentive

3 (IFA) Mention IFA Promise IFA

4 (Clock) Non-monetary clock No Incentive

5 (ATM Card, IFA)$20 ATM Card

Mention IFAPromise IFA

2005 Treatment Groups

Slide 16

Slide 20

Summary of 2005 Face-to-Face Study

As implemented, Prepaid Monetary, “Prepaid” Non-monetary, and Individual Farm Analyses Incentives:

Had no significant impact on response rates, although response rates did go up slightly

Increased costs

It’s possible that a different method of distributing a non-monetary incentive or a different non-monetary incentive all together would have been more effective

Slide 21

Recommendations

Continue the use of monetary incentives for ARMS “Core” mail-out/mail-back data collection methodology and include a control group at least every other year

Do not use prepaid monetary or “prepaid” non-monetary incentives for face-to-face data collections as a means of increasing response rates

Track the card cashing rates of respondents and nonrespondents every year to maintain cost effectiveness of incentive methodology for mail data collection – part of studying the long-term effects on NASS’ entire survey program.

Contacts:

Dan Beckler: dan_beckler@nass.usda.gov

Kathy Ott: kathleen.e.ott@census.gov

Norman Bennett: norman_bennett@nass.usda.gov

Slide 23

References:

Beckler, D. K. Ott, P. Horvath. (2005). Indirect Monetary Incentives for the 2004 ARMS Phase III Core. Research and Development Division Research Report RDD-05-05. National Agricultural Statistics Service, United States Department of Agriculture.

Beckler, D. K. Ott (2005). Indirect Monetary Incentives With a Complex Agricultural Establishment Survey. Proceedings of the Joint Statistical Meetings of the American Statistical Association.

McCarthy, J.S. D. Beckler, K. Ott (2006). The Effect of Incentives on Response in 2005 ARMS Phase III Interviews. Research and Development Division Research Report RDD-06-07. National Agricultural Statistics Service, United States Department of Agriculture.

Slide 24