Basic Research Shapiro 10_2011

8/9/2019 Basic Research Shapiro 10_2011

http://slidepdf.com/reader/full/basic-research-shapiro-102011 1/39

Founder Institute: Basic Research

Dan Shapiro

Ontela



Three projects

• Researching your market with Mturk and

SurveyMonkey

• Testing your company with Google AdWords

• Picking your pitch with Excel



Mturk & Surveymonkey



Surveymonkey

• Cheap tool for building surveys

• You’ll probably need the “pro” subscription

• Zoomerang is even better (but more

expensive)

The problem: you need people to actually take

the survey



mTurk

• Amazon Web Services offering

– Makes it easy to recruit people to do small tasksfor very little money

– Designed to replace AI

– Fund with a regular old Amazon account

– Try it in bits and pieces

•

Perhaps the worst named product, ever – OK, maybe “expertsexchange.com” is worse



What you need to do

• Pay turkers to take surveymonkey surveys



Example surveys

• Demographics, “state of the industry”

• Sales literature fodder

• Usability test

• Naming

• Competitive/landscape research

• Pricing

• Survey your customers (if you’re big)



But is it “valid”?

• Pretty much

• Turkers are reasonably representative of the internet-using public

• Studied pretty thoroughly in academia

•

Empirically our results match internal quantitativeresearch of our customers

• Do you know how shoddy most of the research outthere is? It’s great by comparison

•Recommended reading:

http://bit.ly/7ZhCBs



ENTICING PEOPLE TO DO IT

Step 1



Price it and sell it

• Sell it in the HIT: “Just take a minute!” “Fun survey!” “Getpaid to help a startup!” – Be sure to read the T&Cs about what you can & can’t do

• Price it right – 11 cents is a good starting point

– 21 cents got us 786 responses to a 10 page survey in two weeks – $1.01 really gets people’s attention

– The price controls the rate at which people respond• Start low/slow and increase as needed

• NB: A new HIT means people become re-eligible –

So you can’t combine results originating from different HITs – You can add people to an existing HIT, though, if you want more

data



GET THE RIGHT PEOPLE

Step 2



The secret: throw what you don’t want

• Do not ask, “Are you an asian-american medical doctor licensed topractice in Ohio?” – Ummm... Yes?

– Many turkers will answer in a way that gets them paid

• Collect data about the respondents, then ignore (and pay anyway)those you don’t want – Seems wasteful, but they’re cheap

– Does make this expensive for gathering data on niches

• Some work arounds exist – Gather real contact info in survey one; real compensation comes when

you re-contact them for survey two• Violates TOS

– Use mTurk built-in filtering capaibilities• I have no idea how these work



CONNECTING MTURK AND

SURVEYMONKEY

Step 2



Not too hard

• Build some sort of API fancitude

– I don’t know how to do this

• Put a “code” at the end of the survey

• The HIT is: “Go here, take the survey. It willgive you a password. Enter the password

here:______”

•

Use very explicit instructions – Open a tab, enter this URL... etc



BUILD A GOOD SURVEY

Step 4



“Ask” versus “Tell”

• In “Ask” surveys, the goal is to learn

• In “Tell” surveys, the goal is to create good

marketing material

• You can ask first and tell second in the samesurvey, but not vice versa

Don’t confuse them!!!(i.e. don’t believe your own hype)



Basics

• Ask anything you might ever possibly want

• Get demographic data

– Suggest mirroring the US Census questions

– Lets you slice by demographics

– Lets you compare your results to US averages

• Make almost every question required

–

Or else people can just skip through to the end



Be careful

Things to avoid in “ask” surveys:

• Leading questions

• Order of questions

• Identifying what you’re interested in

•

Telling them who you are – Consider this when creating your account names!

• Creating unrealistic situations – E.g. making them read a page of info when a real customer will

have only a one-line description

Ref: “How to lie with statistics”(Amazon: http://bit.ly/7aot27)



Advanced techniques you should use

• Skip logic

– Annoying but not hard

• Randomized surveys

– Hack combines skip logic with “randomize order of responses”

• Sneaky validity double checks

– Ask the same question a different way

– Leading questions that lead in the wrong direction



ITERATE

Step 5



Do, retry, do again

• Start out with 10 tasks

• Review the results

• Fix the survey

• Repeat• Try again with 100

• Etc.

You will discover and fix all sorts of weird stuff



RUN THE FULL SURVEY

Step 6



Launch the big one

• 1000 respondents gives

you 3% MOE

• Consider subsample

size – if you want to

report on women only,

for example

• Do-overs are not

unaffordable

1,000 * $0.21 = $210



Mind your feedback

• Pay daily

• Watch for user comments; they often find

broken things

– But also are often just confused

• If you get banned, follow the directions to fix

it

– And read the terms more carefully next time!



ANALYZE THE RESULTS

Step 7



Play with the filters

• Exclude unqualified people

• Look at interesting subpopulations

• Check out the boolean combinations

especially



Write up the results

• Parrot the style of analysts

• Use lots of pretty graphs

• Combine related stuff (like multiple questions

in one graph)

• Use filters to expose information



Dan’s todo

• Import in to a database

• Normalize for demographics

• API integration

• Zoomerang

• mTurk qualifications



Google adwords



What’s cool about adwords

• You only pay for success

• You can use it a la carte

• It tests in a crucial, real-world environment

– Assuming you’re going to actually use adwords or

similar to promote your service

• Doing well on adwords lowers your cost of

acquisition



What to use adwords for

• Measuring zeitgeist – Use the tools to see what people are searching for

• Your pitch – Use a placeholder name

– Vary the ad – your “pitch”• Your name/URL

– Once you’ve fixed a pitch, try different names and/orURLs

• Driving traffic to your site to measure stuff – Landing page effectiveness etc.



Considerations

• Your eyeballs are biased

– Your keywords determine who you’re surveying

• Expensive for real customer acquisition

– If a customer is worth $1, forget it

• Slow

• They’re prone to turning your ads off



PICKING YOUR PITCH WITH EXCEL



Step 1

• Brainstorm all the characteristics that are

important

– How pitch performs on Google Adwords

– Maximum revenue earnable

– Time to first revenue

– ...



Step 2

• Weight each factor from Step 1 on a scale of

1-10



Step 3

• List every business concept you can think of



Step 4

• Assign a score for each characteristic



Step 5

• Math time! Multiply each score times weight,

then sort.

Tada! You have your new company.

(not really, but it’s a good start)



[email protected]

email me – any time

Documents

Basic Research Shapiro 10_2011