DRAFT - not for publication Nonresponse Bias Analysis in a Survey of Banks Carl Ramirez U.S. Government Accountability Office [email protected]

DRAFT - not for publication

Nonresponse Bias Analysis in a Survey of Banks

Carl Ramirez

U.S. Government Accountability [email protected]


Overview

• Describe NR bias analysis techniques and results in one survey of small, minority-owned financial institutions.

• Make observations about NR bias analyses in small population surveys

• Describe emerging NR bias analysis practices at one US Federal research agency


Survey of 196 Minority-Owned Banks

• Census of all institutions defined by at least one regulator as minority-owned as of March 2006.

• Web survey, mail/fax options upon request• President/CEO was targeted respondent• 76% unit response rate (AAPOR RR2).• Key estimates: Awareness of and attendance at

regulator programs, rating of regulator efforts to preserve minority ownership, financial outlook


Survey Fieldwork

• Field period: March 14 to April 28 – 6 weeks & 3 days• Contacts:

– Precontact call in Feb. to determine eligibility and get email

– Prenotification email in early March

– Survey cover email March 14

– 2 email NR followups

– National Bankers Association endorsement contacts

– Multiple phone NR followups by program analysts

– Final reminder and closeout emails in late April


NR Bias Analysis Typology(Groves & Brick)

• Compare survey estimates to other benchmarks

• Compare R’s to NR’s on auxiliary variables

(frame, external data, fieldwork observations, etc.)

• Examine variation within respondents (subgroups,

converted nonrespondents, early/late responders)

• Alter weighting adjustment


Unique Aspects of Establishment Survey NR Bias Analyses

• More longitudinal sample designs = previous

response benchmark opportunities

• Richer administrative data = more benchmarks

and survey variable correlates

• Smaller populations and subgroups = potential

for examining individual nonrespondents


NR Bias Evaluation Methods Applied to Survey of Banks

• Compare respondents to sample on key variables

• Compare response rates of subgroups defined by frame

information, related to survey variables of interest

• Level of Effort (Time of Return)

• Subsample of nonrespondents converted with high-effort

followup


Respondents vs. Entire Sample on key auxiliary variable: Regulator

R’s Sample

FDIC 54% 55%

Federal Reserve 11 11

OCC 23 22

OTS 12 11


Respondents vs. Sample: Minority Type

R’s Sample

African American 26% 24

Native American 10 10

Asian 39 34

Hispanic 17 24


Respondents vs. Sample: Size

R’s Sample

< $100 Million assets 43% 41%

$101-$300 M 32 32

$301-$500 M 12 9

$501M - $10 billion 11 15


Differences in Response Rate by Regulator of Bank

R’s NR’s

Federal Reserve 82% 18

OTS 82% 18

OCC 77% 23

FDIC 73% 27


Differences in Response Rate by Minority Type

R’s NR’s

African American 81% 19

Native American 80% 20

Asian 76% 24

Hispanic 70% 30


Differences in Response Rate by Profitability

R’s NR’s

Low return on assets 81% 19

Medium 75% 25

High 73% 27


Level of Effort Analysis(Time of Return)

Key Estimates Early

(n=51)

Middle

(84)

Late

(9)

Received regulator mailings on minority program

48% 50 56

Aware of FDIC minority bank web page

55% 51 44

Attended FDIC minority roundtable or conference

45% 43 56

Used regulator’s technical assistance program

25% 27 56


Analysis of Respondents with other NR Characteristics

• Comparison of low-salience (low response

propensity) respondents to all other respondents:

12 banks not considered by primary regulator as

“minority owned” but so designated by one of

the other 3 regulators.

• Removing these respondents changed key

estimates by 1-2% , none by more than 4-5%


Micro-Level Examination of Nonrespondents

• Small sample allowed in-depth study of

individual banks not responding

• 4 explicit refusals – known reasons appeared

largely unrelated to key measures

• Personal contacts for NR followup yields known

reason for delay


Emerging GAO practice on NR Bias Analysis?

• Groves & Brick typology

• Suggests hierarchy of methods on 2 dimensions:

– Prefer use of data related to survey variable over

comparison of subgroups or time of return analysis

– Prefer methods using data available for entire

population over portion of population or sample

• If conduct analyses, describe in report

Documents

DRAFT - not for publication Nonresponse Bias Analysis in a Survey of Banks Carl Ramirez U.S. Government Accountability Office [email protected]