View
214
Download
0
Tags:
Embed Size (px)
Citation preview
1
Randomized Control Trials for Business Solutions:
Putting Them to Work for You
Jon ZinmanDartmouth College
October 6, 2006
3
Overview of Talk
1. Examples of product spaces where academics can help CUs improve profits and member satisfaction (using randomized-control trials)
– Focus on mortgages, small business lending• Where $ is!• Biggest opportunity for comparative advantage. Little
academic and consultant attention compared to, e.g., saving, financial planning, credit card use
2. Methodology:– How we do it– Operational requirements
4
Example 1: Mortgage Menus
• Too much choice depresses consumer demand– Psychology studies– Bertrand-Karlan-Mullainathan-Shafir-Zinman (2005)
showed this in an experiment with an actual consumer finance company
• But what’s the “right” menu of choices?– “Right” depends on our partner institutions’ objectives,
but for now let’s say:• Profitable• Sustainable (long-run member satisfaction)
5
Example 1: Mortgage Menus
• Here is a hard choice:– Choice A: 7-1 ARM, no points at 7.5% vs. – Choice B: Fixed rate, no points, at 8%.
• However, if you add choice C,• Choice C: Fixed rate, 2 points, at 8.05%
• B looks much more attractive and becomes easy choice
6
Outline of a Mortgage Menus Experiment
Experiment:
1) Identify small set of promising menus
2) Run horserace in course of actual ops
3) Analyze results and identify best menu(s)
7
Example 2: Refinancing
• Evidence suggests that mortgagees procrastinate or don’t pay attention, therefore don’t finance at right time
• Business implication. Help members refi at right time and: – Profit (new clients, member retention, more
frequent refis)– Increase client satisfaction
8
Outline of a Refi Experiment
Marketing for attention.• Small cues, messages drive clients choices• Our prior work shows this. Showing a female
photo on direct mail increased consumer loan takeup among dormant male clients by about same amount as a 50% drop in the interest rate
• Tap social networks (potential comparative advantage for CUs)
• So test a set of promising cues/messages via direct mail, phone
9
Outline of Another Refi Experiment
Experiment with automatic refis.– Help members choose refi triggers in advance
• comparative advantages here for CUs: trust, personal touch
– Offer binding and non-binding commitments– Test takeup rate, profitability, ex-post member
satisfaction for a few automatic refi options
10
Example 3:Home equity loan/line utilization
• Seems low given:– Simultaneous holdings with relatively
expensive credit card debt– # of homeowners who have substantial card
debt and no HELOC or HEloan
11
Outline of Home Equity Experiment
• Use psychological principles to drive use of equity lines (instead of more expensive forms of borrowing)
• Marketing with:– Salient rate comparisons– Cues (picture of family, home improvements)– Make mental accounting work for HELOCs
(“put your interest savings back into your house, kids’ education, etc.”)
12
Example 4: Pricing Riskon Small Business Loans
• How do you know where to draw the line?• Some lenders leave profitable deals on table.
Example from Karlan-Zinman (2006):– Experiment on high-risk consumer loans– Added step to Lender’s normal underwriting process:
applicants who were rejected but close to the bar got a loan 25% or 50% of the time (randomly assigned)
– These loans ended up being profitable– Also measurably positive impacts on clients
• Another method we use for identifying profitability frontier: credit scoring for small businesses
13
What we Mean by “Experiment”:Randomized-Control Trials (RCTs)
• What they are:– Identical members or prospective clients get
different offers– Offers are randomly assigned (“drawn out of a
hat”) within a reasonable, predefined range• range is guided by business considerations
14
Why RCTs?
• RCTs are “gold standard” method for identifying causality in social, medical science; biz applications
• Highly practical/profitable biz proposition– Our work with smaller firms– Larger firms have integrated RCTs into routine
marketing and merchandizing operations, face-to-face contact
• Credit card companies• Amazon• Food/drug retailers• H&R Block
15
RCTs: Step-by-Step1. Identify product space and key questions
2. Design experiment to answer key questions
3. Implement design into operations• Piloting• Full launch
4. Monitor implementation for compliance with design
5. Analyze data from experiment– Data sharing: consent and disclosure issues
6. Apply results of experiment
16
RCT Step #1:Identify Area(s) for Experimentation • Process: focused brainstorming
• Players: researchers, senior mgmt.
• Outcomes:– choice of product spaces– outlines of experimental design like ones I
gave earlier
17
RCT Step #2.Design experiment
• Process: take design outline, meld it to operational realities of firm.• Players: research team, senior mgmt, field/area managers, front-line
personnel, IT personnel• Examples:
– Ensure data sharing feasible: consent, disclosures required?– Designing direct mailers– Coordinating experimental direct mailers with routine marketing efforts– Training loan officers to offer a new automatic refi product. Say some clients will
get direct mail or face-to-face offer of this new product, other won’t. Then loan officers need to know:
• Content of new product(s)• Presentation• Who gets the offer? (compliance with randomization)
– Usually involves a software add-on to MIS that loan officer would reference for each client, and automatically prompt a new product offer, or regular product offer
• Staffing requirements:– Research team: we bring a project manager– CU: may want dedicated project manager (depends on scale)
18
RCT Step #3:Implement design into operations
3a. Pilot experiment
• Small scale
• Work out kinks:– Design tweaks– Operational tweaks (compliance, clunkiness)– Fix/eliminate backfires (member complaints,
staff burdens)
3b. Full launch
19
RCT Step #4:Monitor Compliance with Design
• Audits/site visits for any face-to-face components
• Statistical tracking
20
RCT Step #5:Analyze Data from Experiment
• Researchers generate a report– Upfront issue: make sure can hand
researchers the requisite data needed for analysis
• What consent, disclosures are needed, if any?
21
Step #6:Apply Results of Experiment
• What have we learned, and what should the CU do next?– Discussions between researchers, mgmt– Lead to recommendations from researchers
about how to apply the results:• Specific product, pricing, marketing decisions• Strategy and positioning• Opportunities for additional experiments