View
4.710
Download
3
Category
Preview:
DESCRIPTION
Video available at: http://www.youtube.com/watch?v=i8MGTZ3KWmc At April’s @IndeedEng Talk we introduced Indeed’s philosophy and practice of A/B testing. In this talk, two Indeed product managers will discuss how we used data-driven opportunity analysis and iterative testing to build two products. From product vision to product success, we’ll describe what we tested, how it performed, and what we learned from it. Product managers, designers, and engineers who want to learn how to prioritize product and feature ideas, iterate through tests, or optimize a funnel will find valuable insights to apply to their own products. Graham Davis is a Senior Product Manager for Employer Products at Indeed. Prior to Indeed, Graham previously worked for several startups and got an MBA from Harvard Business School. Donald Wysocki is Product Director for Job Search at Indeed. Prior to Indeed, Donald worked at frog design and Microsoft.
Citation preview
Graham Davis Donald WysockiSr. Product Manager Product Director
Diving Deeper into Data-Driven
Product Design
I help people get jobs.
Our guiding principle
Do what's best for job seekers
Simple
Fast
Comprehensive
Relevant
Data-Driven Product Design
1. Vision
Data-Driven Product Design
Data-Driven Product Design
1. Vision2. V1
Data-Driven Product Design
1. Vision2. V13. Test and measure
Data-Driven Product Design
1. Vision2. V13. Test and measure4. and test and measure and...
Company Reviews
Graham Davis
Indeed Apply
In the past 12 months...
500 million jobs on Indeed
39 billion job searches
6 billion jobs viewed
Estimated720 million applications
~20 minutes to apply
10,000,000 days
How it works
Applications
Interested parties, please email your resume with the subject line "PHP Web Developer" tojobs@smallworldlabs.com
Applications
● Email● Simple resume upload
Applications
● Email● Simple resume upload● Account + complex forms
Applications
● Email● Simple resume upload● Account + complex forms● Mobile
Applications
● Email● Simple resume upload● Account + complex forms● Mobile
Applications
● Email● Simple resume upload● Account + complex forms● Mobile
Easily apply to all jobs on all devices.
Vision
So, where do we start?
Nearly every job application was completed
on another site
2,000 jobs posted to Indeed by small businesses
V1: How can we make it easier?
What if job seekers who previously applied could re-
use their resume?
-10%
0%
Why?
Maybe job seekers didn't understand they have a saved
resume
Can an Indeed Resume make it easier to apply?
Applications Per Week+0%
For job seekers who are signed in & have an Indeed
Resume, the application rate increased by 2x
Promising, but...
2x a small number is still a small number
We need more jobs
Opportunities
Applications
● Email● Simple resume upload● Mobile
Applications
● Email● Simple resume upload● Mobile
5,000
Postedon Indeed
9,500Jobs
Applywith Email
A new test:1. US only2. Signed-in3. Indeed Resume
Applications Per Week+33%
Can we make this available to more job seekers?
Expanded test:1. US only2. Signed-in3. Indeed Resume
Total Job Applications
+70%
Can we get more jobs?
INTL14,500
US
207,000Expand Internationally
Applications per Week
+58%
Can we get even more jobs?
Applications
● Email● Simple resume upload● Mobile
Applications
● Email● Simple resume upload● Mobile
Apply on the original website
Trusted,Everywhere
First (and fast) partner test:ZipRecruiter
Applications from Indeed+109%
More applications from the same number of job seekers
How can we make it easier for job seekers to find these
jobs?
+2% applications
Does branding actually help job seekers?
+6% applications
Progress:● Easier apply process● More jobs● Easier to find
Applications
● Email● Simple resume upload● Mobile
Applications
● Email● Simple resume upload● Mobile
The big win...
WebMobile
> 1/3 of searches
<5% of the Fortune 500 accept applications on
their mobile sites
3rd party applymobile apply
more resumesmore posted jobs...
Applications per Week
+584%
Data driven design isn't just about numbers and charts...
I help people get jobs.
Company Reviews
Graham Davis
I help people get jobs.
I help people get jobs.
the best<
What is it like to work at...
typical day
typical daymanagement
typical daymanagementco-workers
647
important decision
opportunity to help job seekers
Help job seekers find the best employers.
Vision
3,000,000
0
Jan 2011: V1
Not perfect
We don't know anything yet
Good enough to start testing
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
Remove anything that's not absolutely necessary
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
● "write a review" button● review form● search for companies● see reviews on company page● see reviews on job pages● see reviews on search results● best places to work● filter by location● filter by job function● review commenting● review voting● featured reviews
Built it
August 2011: 8,000
August 2011: 8,000only 0.2% of jobs
August 2011: 8,000only 0.2% of jobs
1 of every 50 search pages
August 2011: 8,000only 0.2% of jobs
1 of every 50 search pagesNOT good enough
Goal: every search page
500,000
300
300
10,000
300
10,000
+3000%
So we started testing
Why aren't people submitting reviews?
How can we make it easier?
What can we learn from successful review sites?
+27%387
What can the data tell us?
How can we improve company search?
-45%
Step back and assess
What works for one site does not necessarily work
for another site
Indeed is not a reviews site
Indeed is a job site
How can we engage our job seekers where they are?
The flow: Oct 2011
What can the data tell us?
2 ways to improve a funnel
2 ways to improve a funnelImprove conversion at step
2 ways to improve a funnelImprove conversion at step
Remove step entirely
Can we remove steps?
Yes,if we target job seekers with
Indeed resumes.
New flow: Nov 2011
+343%
1,507
What can the data tell us?
Avg reviews per reviewer: 1
Avg reviews per reviewer: 1Avg jobs per resume: 3.6
How can we increase reviews per reviewer?
Before
After
+46%3,719
Avg reviews per reviewer: 1Avg jobs per resume: 3.6
1.5
Avg reviews per reviewer: 1Avg jobs per resume: 3.6
why not 3.6?1.5
What can the data tell us?
Indeed
Zoominfo
?42%
Indeed
Zoominfo
My Perfect Gig
+57%5,834
What would happen if we had more targeted
promos for job seekers with resumes?
+36%7,919
Can we remove more steps?
Before
After
+104%
13,010
Can we remove anything from the review form?
RatingsReview TitleReviewProsConsJob TitleJob LocationStart/End Date
RatingsReview TitleReviewProsConsJob TitleJob LocationStart/End Date
Resume
+24%17,168
Today...
3,000,000
1.7M companies
60,000 / week
60% of jobs
6 out of 10
we moved the needle more by removing than
adding
Simple
Perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away.
Antoine de Saint Exupéry
Donald Wysockidonald@indeed.com
Graham Davisgdavis@indeed.com
Q&A
Next @IndeedEng TalkEngineering Velocity: Building GreatSoftware Through Fast Iteration
Dan Heller, Engineering Manager
7pm Wednesday, August 28th
Recommended