1

Auditing Section Doctoral ConsortiumAuditing Section Doctoral Consortium2006 Auditing Section Midyear Conference2006 Auditing Section Midyear Conference

January 2006January 2006

Linda McDanielLinda McDanielUniversity of KentuckyUniversity of Kentucky

Experimental Research in Assurance & Auditing

2

Acknowledgements

Many colleagues, but particularly Jane Kennedy Bill Kinney Laureen Maines Mark Peecher

3

Experimental Research Session: Objectives

■ Review strengths and weaknesses of experimental research

■ Discuss how to capitalize on these strengths (i.e., summarize elements of good experimental design)

■ Illustrate with an example arising from SOX reguations

4

Experimental Research is … Systematic, controlled, empiricalcontrolled, empirical, critical

investigation of phenomena guided by guided by theorytheory and hypotheses about the presumed relations among such phenomena (Kerlinger)

characterized by, active manipulation of variables of

interest to generate new data the random assignment of

participants to specified conditions

controllcontrolled,ed,

presumedpresumedrelationsrelationstheotheoryry

guided guided byby

5

Comparative Advantages

Ability to test causal relations, not just associations Manipulate variables of interest

Internal validity Control/randomize effects of other

variables Disentangle variables confounded in

natural setting

6

Timeliness: no need to wait on the real world to create data

See McDaniel and Hand (CAR, 1996)

Ex ante research is possible Conditions that do not exist in natural

settings can be created in the lab Gaynor, McDaniel, and Neal (TAR,

forthcoming) Hirst and Hopkins (JAR, 1998)

Comparative Advantages

7

Comparative Advantages

Examination of sub-judgments (determinants of decisions) and processes

Kadous, Kennedy, and Peecher (TAR, 2003) Hoffman, Joe, and Moser (AOS, 2003) Maines and McDaniel (TAR, 2000)

Thus, experiments can answer Thus, experiments can answer how, how, when, and whywhen, and why important features of important features of the accounting process and the accounting process and environment influence environment influence behaviorbehavior as as well as decisionswell as decisions

8

Relative Disadvantages

External validity Task is abstraction from real world Variables manipulated at discrete levels Participants may not be representative

Small sample size Limited access to participants for real-world,

complex auditing/accounting issues

Reduced ability to replicate No second chances (without significant costs)

9

Designing an Experiment

After the researcher identifies an interesting, relevant question that calls for an experiment … (i.e., post Kinney 3 paragraphs)

… he/she must develop an effective (good) research design, i.e., to draw causal inferences Use theory to guide predictions

““It is the theory that decides It is the theory that decides what can be observed.”what can be observed.” Albert Einstein

Minimize threats to construct, internal, and statistical validity

10

Vs and Zs Prior-influence & contemporaneous factors (alternative explanations)

4. Statisti

cal Validity

Operational Definition

X

Operational Definition Y

Operational

Libby et al. (2002) Predictive Validity Framework

Concept X

Concept Y

Conceptual

Independent (X) Dependent (Y)

1. Theory

2. Construct Validity 3.

5. Internal Validity

11

Designing an Experiment

What variables will you manipulate?

Number & levels of independent variables

Interactions?Libby et al. (2002) Predictive Validity Framework

12

Designing an Experiment

What variables will you control? (internal validity) Account for Vs and Zs (see Kinney (TAR, 1986))

by Random assignment of participants

Hold variables constant by design/ selection (within-participant design; match on Vs)

Measure covariates / statistically remove effects (e.g., covariate analysis, regression)

13

Other Necessary Design Choices

Professional participants?

Incentives?

Within- versus between- participants design?

See Libby, Bloomfield, and Nelson (AOS, 2002) for a discussion of each

14

Professional Participants?

Theory should dictate this choice Libby and Kinney (TAR, 2000) Maines and McDaniel (TAR, 2000) See also Libby and Luft (AOS, 1993)

Professionals are a limited resource

Libby, Bloomfield & Nelson (AOS, 2002)

Professionals exhibit stronger selection bias relative to non-professional groups

Peecher & Solomon (IJA, 2001)

15

Incentives?

Why? “…no skin in the game…”

When? Camerer & Hogart (JRU, 1999)

16

Within- versus Between- Participants?

Enhanced statistical power as participants serve as their own control

See Schepanski, Tubbs, and Grimlund (JAL, 1992)

Increased salience of treatment effects

Vulnerability to carry-over effects

Requires proper statistical analysis

17

Turning Observations into a Researchable Question

Do the new SOX regulations related to NAS result in improved audit quality?

Why is this interesting or important?

How can we examine?

18

Real World Problem and Regulatory Actions After corporate abuses, SEC seeks to ban

all auditor-provided NAS Concerns about auditor independence, audit

quality, and investor confidence

Conceding certain NAS improve audit quality, SEC limits NAS auditors can provide to clients but requires ACs to pre-approve services after considering

auditor independence and audit quality Registrants to disclose AC pre-approvals and

fees paid to auditor (by category)

19

An Example

The Effects of Joint Provision and Disclosures of Non-audit

Services (NAS) on Audit Committee (AC) Decisions and

Investor Preferences

20

Theory

Pre-approval process makes ACs directly accountable to 3rd parties for auditor independence and audit quality

Disclosures (of pre-approvals and audit fees) makes ACs publicly accountable to investors for perceived independence

anecdotal reports suggest ACs are avoiding allowable NAS

21

Predictions / Research Hypotheses

ACs will be more likely to recommend joint

provision when the NAS improves audit quality (AQ)

less likely to recommend joint provision when public disclosures are required

The disclosure effect holds even when ACs believe joint provision improves AQ

22

NAS/AQ relation & required public

disclosures

Predictive Validity Framework

Conceptual

Independent (X) Dependent (Y)

Account-ability

ACs pre-approval decisions

Type of NAS and type of company

Joint provision recommendatio

n

Vs and Zs Experience with NAS approval; beliefs about effects of NAS on auditor

independence and synergies with audit; audit experience, etc.

Operational

23

Operational Independent Variables

Type of Service: Effect of NAS on Audit Type of Service: Effect of NAS on Audit QualityQuality

Risk Management ServicesRisk Management Services

Joint provision improves audit quality

Human Resource ServicesHuman Resource Services

Joint provision has no effect on audit quality

24

Measured Independent Variable

Measured Independent Measured Independent Variable:Variable:

Belief about NAS and audit quality relation

See Libby et al. (AOS, 2002) for when this approach is justified and implications for interpretation

25

Operational Independent Variables

Type of Company: Disclosure Type of Company: Disclosure RequirementRequirement

Publicly-traded companyPublicly-traded company

Company is required to make public disclosures

Privately-held companyPrivately-held company

Company is not required to make public disclosures

26

Operational Dependent Variableand Controls

Measured Dependent Measured Dependent Variable: Variable:

Joint provision recommendation Reasons for and against firm selection

Manipulation Checks / Control Manipulation Checks / Control Variables: Variables:

Audit quality manipulation checkNAS quality by provider Effect of joint provision on auditor objectivityDemographic information

27

Participants & Other Choices

Participants were Corporate Directors attending a KPMG Audit Committee Institute Roundtable

No monetary incentives

Between-participants design

28

Some Lessons Learned

1. Work on projects that really interest you and for which you have a comparative advantage!

2. Good experimental papers require a lot of up-front time and effort. This pays off!

3. Always:Write Kinney 3 paragraphs / have others

review!Prepare Libby boxesPilot test (as many times as necessary)Share with colleagues often throughout the

process