121
Rand Fishkin, Wizard of Moz | @ randfish | [email protected] Why Great Marketers Must Be Great Skeptics

Why Great Marketers Must Be Great Skeptics

Embed Size (px)

DESCRIPTION

Rand Fishkin's presentation on the application of skepticism in web marketing.

Citation preview

Page 1: Why Great Marketers Must Be Great Skeptics

Rand Fishkin, Wizard of Moz | @randfish | [email protected]

Why Great MarketersMust Be Great Skeptics

Page 2: Why Great Marketers Must Be Great Skeptics

This Presentation Is Online Here:

bit.ly/mozskeptics

Page 3: Why Great Marketers Must Be Great Skeptics

Great SkepticismDefining

Page 4: Why Great Marketers Must Be Great Skeptics

I have some depressing news…

Page 5: Why Great Marketers Must Be Great Skeptics
Page 6: Why Great Marketers Must Be Great Skeptics
Page 7: Why Great Marketers Must Be Great Skeptics

Does anyone in this room believe that the Earth doesn’t revolve around the Sun?

Page 8: Why Great Marketers Must Be Great Skeptics
Page 9: Why Great Marketers Must Be Great Skeptics

The Earth (and everything in the solar system,

including the Sun) revolves around our system’s

gravitational Barycenter, which is only sometimes

near the center of the Sun.

Page 10: Why Great Marketers Must Be Great Skeptics

Let’s try a moremarketing-centric example...

Page 11: Why Great Marketers Must Be Great Skeptics

In 2009, Conversion Rate Experts built us a new landing page, and

increased our subscribers by nearly

25%. What did they do?

Via CRE’s Case Study

Page 12: Why Great Marketers Must Be Great Skeptics

One of the most commonly cited facts

about CRE’s work is the “long landing page.”

Page 13: Why Great Marketers Must Be Great Skeptics

The Crap Skeptic The Good Skeptic The Great Skeptic

Let’s change our landing page to be a long one

right now!

We should A/B test a long

landing page in our conversion

funnel.

How do we know page length was

responsible? What else changed?

Page 14: Why Great Marketers Must Be Great Skeptics

The Crap Skeptic The Good Skeptic The Great Skeptic

“I do believe sadly it’s going to take some diseases coming back to realize that we need to change and develop vaccines that are safe.”

“Listen, all magic is scientific principals presented like "mystical hoodoo" which is fun, but it's sort of irresponsible.”

"The good thing about science is that it's true whether or not you believe in it." 

Page 15: Why Great Marketers Must Be Great Skeptics

In fact, we’ve changed our landing pages numerous

times to shorter versions and seen equal success. Length, it

would seem, was not the primary factor in this page’s

success.

Page 16: Why Great Marketers Must Be Great Skeptics

What separates the crap, good, & great?

Page 17: Why Great Marketers Must Be Great Skeptics

Assumes one belief-reinforcing data point is evidence enough

Doesn’t question what’s truly causal vs. merely correlated

Doesn’t seek to validate

Page 18: Why Great Marketers Must Be Great Skeptics

Doesn’t make assumptions about why a result occurred

Knows that correlation isn’t necessarily causal

Validates assumptions w/ data

Page 19: Why Great Marketers Must Be Great Skeptics

Seeks to discover the reasons underlying the results

Knows that correlationdoesn’t imply causality

Thoroughly validates, but doesn’t let imperfect knowledge stop

progress

Page 20: Why Great Marketers Must Be Great Skeptics

Will more conversion tests lead to better results?

Testing

Page 21: Why Great Marketers Must Be Great Skeptics

Obviously the more tests we run, the better we can

optimize our pages. We need to build a “culture of

testing” around here.

Page 22: Why Great Marketers Must Be Great Skeptics

Via Wordstream’s What is a Good Conversion Rate?

Page 23: Why Great Marketers Must Be Great Skeptics

Via Wordstream’s What is a Good Conversion Rate?

Do Those Who Test More Really Perform Better?

Page 24: Why Great Marketers Must Be Great Skeptics

Hmm… There’s no correlation between those who run more tests across more pages and

those who have higher conversion rates. Maybe the number of tests isn’t the right

goal.

Page 26: Why Great Marketers Must Be Great Skeptics

Trust

Word of Mouth

LikabilityDesign

Associations

Word of Mouth

Amount of Pain

CTAs

UX

Effort Required

Process

Historical Experiences

Social Proof

Copywriting

CONVERSION DECISION

Timing

Discovery Path

Branding

Price(it’s a complex process)

Page 27: Why Great Marketers Must Be Great Skeptics

How do we know where ourconversion problems lie?

Page 28: Why Great Marketers Must Be Great Skeptics

Ask Smart Questions to the Right PeoplePotential Customers

Who Didn’t BuyThose Who Tried/Bought But

Didn’t Love ItCustomers Who

Bought & Loved It

Professional, demographic, & psychographic characteristics

Professional, demographic, & psychographic characteristics

Professional, demographic, & psychographic characteristics

What objections did you have to buying?

What objections did you have; how did you overcome them?

What objections did you overcome; how?What would have made you

stay/love the product?What would have made

you overcome them?

What do you love most? Can we share?

Page 29: Why Great Marketers Must Be Great Skeptics

We can start by targeting the right kinds of

customers. Trying to please everyone is a recipe for

disaster.

Page 30: Why Great Marketers Must Be Great Skeptics

Our tests should be focused around overcoming the

objections of the people who best match our

customer profiles

Page 31: Why Great Marketers Must Be Great Skeptics

Testing button colors

Page 32: Why Great Marketers Must Be Great Skeptics

Testing headlines, copy, visuals, & form fields

Page 33: Why Great Marketers Must Be Great Skeptics

Designing for how customers think about their problems &

your solution

Page 34: Why Great Marketers Must Be Great Skeptics

THIS!

Page 35: Why Great Marketers Must Be Great Skeptics

Does telling users we encrypt data scare them?

Security

Page 38: Why Great Marketers Must Be Great Skeptics

Via Visual Website Optimizer

A/B Test Results

They found that without the secure icon had over 400% improvement on conversions as compared to having the image.

[Note: results ARE statistically significant]

Page 39: Why Great Marketers Must Be Great Skeptics

We need to remove the security messages on our

site ASAP!

Page 40: Why Great Marketers Must Be Great Skeptics

We should test this.

Page 41: Why Great Marketers Must Be Great Skeptics

Is this the most meaningful test we can perform right now?

(I’m not saying it isn’t, just that we should prioritize intelligently)

Page 42: Why Great Marketers Must Be Great Skeptics

Via Kayak’s Most Interesting A/B Test

vs.

Page 43: Why Great Marketers Must Be Great Skeptics

Via Kayak’s Most Interesting A/B Test

A/B Test Results

“So we decided to do our own experiment about this and we actually found the opposite that when we removed the messaging, people tended to book less.”

- Vinayak Ranade, Director of Engineering for Mobile, KAYAK

Page 44: Why Great Marketers Must Be Great Skeptics

Good thing we tested!

Good thing we tested!

Your evidence is no match for my

ignorance!

Page 45: Why Great Marketers Must Be Great Skeptics

What should we expect from sharing our content on social media?

Social CTR

Page 46: Why Great Marketers Must Be Great Skeptics

Just find the average social CTRs and then try to match

them or do better. No brainer.

Page 49: Why Great Marketers Must Be Great Skeptics
Page 50: Why Great Marketers Must Be Great Skeptics
Page 51: Why Great Marketers Must Be Great Skeptics

306/701 = 43.6%... WTF??

Page 52: Why Great Marketers Must Be Great Skeptics
Page 53: Why Great Marketers Must Be Great Skeptics

Phew! We’re not alone.

Via Chartbeat

Page 54: Why Great Marketers Must Be Great Skeptics

Assuming social metrics and engagement correlate was a flawed assumption. We need

to find a better way to measure and improve social sharing.

Page 55: Why Great Marketers Must Be Great Skeptics
Page 56: Why Great Marketers Must Be Great Skeptics
Page 57: Why Great Marketers Must Be Great Skeptics
Page 58: Why Great Marketers Must Be Great Skeptics

OK. We can create some benchmarks based on these numbers and their averages,

then work to improve them over time.

Page 59: Why Great Marketers Must Be Great Skeptics

That is an insane amount of variability!

Page 60: Why Great Marketers Must Be Great Skeptics

There are other factors at work here. We need to understand them before we can create

smart metrics or useful expectations

Page 61: Why Great Marketers Must Be Great Skeptics

Timing

Source

Audience Affinity

Formatting

Network-Created Limitations to Visibility

Brand Reach

Traffic

Engagement

Page 62: Why Great Marketers Must Be Great Skeptics

Let’s start by examining the data and impacts of timing.

Page 63: Why Great Marketers Must Be Great Skeptics

Via Facebook Insights

Page 64: Why Great Marketers Must Be Great Skeptics

Via Followerwonk

Page 65: Why Great Marketers Must Be Great Skeptics

Via Google Analytics

Page 66: Why Great Marketers Must Be Great Skeptics

There’s a lot of nuance, but we can certainly see how

messages sent at certain times reach different sizes and

populations of our audience.

Page 67: Why Great Marketers Must Be Great Skeptics

Comparing a tweet or share sent at 9am Pacific against tweets

and shares sent at 11pm Pacific will give us misleading data.

Page 68: Why Great Marketers Must Be Great Skeptics

But, we now know three things:

#1 - When our audience is online

#2 – Sharing just once is suboptimal

#3 – To be a great skeptic (and marketer), we should attempt to understand each of these inputs with similar rigorousness

Page 69: Why Great Marketers Must Be Great Skeptics

Do they work? Can we make them more effective?

Share Buttons

Page 70: Why Great Marketers Must Be Great Skeptics

After relentless testing, OKTrends found that the following share buttons worked best:

Page 71: Why Great Marketers Must Be Great Skeptics
Page 72: Why Great Marketers Must Be Great Skeptics
Page 73: Why Great Marketers Must Be Great Skeptics

OKTrends found that removing all but a single button (the “like” on Facebook) had

the most positive effect.

Page 74: Why Great Marketers Must Be Great Skeptics

And that waiting until the visitor had scrolled to the bottom of the article produced the

highest number of actions

Page 75: Why Great Marketers Must Be Great Skeptics

We should remove all our social sharing buttons and replace them with a single slide-over social CTA for

Facebook likes!

Page 76: Why Great Marketers Must Be Great Skeptics

Buzzfeed has also done a tremendous amount of social button testing & optimization…

Page 77: Why Great Marketers Must Be Great Skeptics

And sometimes they do this…

Page 78: Why Great Marketers Must Be Great Skeptics

And sometimes this…

Page 79: Why Great Marketers Must Be Great Skeptics

Is Buzzfeed still in testing mode?

Page 80: Why Great Marketers Must Be Great Skeptics

Nope.They’ve found it’s best to show different buttons based on both the type of content and how you

reached the site.

Page 81: Why Great Marketers Must Be Great Skeptics

OK… Well, then let’s do that… Do it now!

Page 82: Why Great Marketers Must Be Great Skeptics

Testing a small number of the most impactful social button

changes should produce enough evidence to give us a direction

to pursue.

Page 83: Why Great Marketers Must Be Great Skeptics

Buzzfeed & OKTrends share several unique qualities:1) They have huge amounts of

social traffic2) Social shares are integral to their

business model3) The content they create is

optimized for social sharing

Page 84: Why Great Marketers Must Be Great Skeptics

Unless we also fit a number of these criteria, I have to ask again:

Is this the most meaningful test we can perform right now?

Page 85: Why Great Marketers Must Be Great Skeptics

BTW – it is true that testing social buttons can coincide with a lot of other tests (since it’s on content

vs. the funnel), but dev resources and marketing bandwidth probably are not infinite

Page 86: Why Great Marketers Must Be Great Skeptics

Does it still work better than standard link text?

Anchor Text

Page 87: Why Great Marketers Must Be Great Skeptics

Psh. Anchor text links obviously work. Otherwise

Google wouldn’t be penalizing all these sites for

getting them.

Page 88: Why Great Marketers Must Be Great Skeptics

It has been a while since we’ve seen a public test of anchor text. And there’s no way to know for

sure how powerful it still is.

Page 89: Why Great Marketers Must Be Great Skeptics

Testing in Google is very, very hard. There’s so many confounding

variables – we’d have to choose our criteria carefully and repeat the test multiple times to feel confident of

any result.

Page 90: Why Great Marketers Must Be Great Skeptics

1) Three word, informational keyword phrase with relatively light competition and stable rankings

Test Conditions:

2) We selected two results (“A” and “B”), ranking #13 (“A”) and #20 ( “B”) in logged-out, non-personalized results

3) We pointed links from 20 pages on 20 unique, high-DA, high-trust, off-topic sites at both “A” and “B”

Page 91: Why Great Marketers Must Be Great Skeptics

A) We pointed 20 links from 20 domains at this result with anchor text exactly matching the query

phrase

#11

#12

#13

#14

#15

#16

#17

#18

#19

#20

B) We pointed 20 links from the same 20 pages as “A” to this URL

with anchor text that did not contain any words in the query

Page 92: Why Great Marketers Must Be Great Skeptics

#11

#12

#13

#14

#15

#16

#17

#18

#19

#20

#1

#2

#3

#4

#5

#6

#7

#8

#9

#10

After 20 days, all of the links had been indexed by Google. “A” and “B” both moved up 4 positions. None of the other results moved more than 2

positions.

Page 93: Why Great Marketers Must Be Great Skeptics

See? Told you it works.

Page 94: Why Great Marketers Must Be Great Skeptics

While both results moved up the same number of positions, it’s

almost certainly the case that #13 to #9 was against more serious

challengers, and thus anchor text would seem to make a difference. That said, I’d want to repeat this

a few times.

Page 95: Why Great Marketers Must Be Great Skeptics

Princess Bubblegum and I are in agreement. We should do the test at least 2-3 more times keeping as

many variables as possible the same.

Page 96: Why Great Marketers Must Be Great Skeptics

1) Three word, informational keyword phrase with relatively light competition and stable rankings

Early Results from a Second Test:

2) We selected two results (“A” and “B”), ranking #20 (“A”) and #14 ( “B”) in logged-out, non-personalized results

3) We pointed links from 20 pages on 20 unique, high-DA, high-trust, off-topic sites at both “A” and “B”

Page 97: Why Great Marketers Must Be Great Skeptics

B) We pointed 20 links from 20 domains to this URL with anchor

text that did not contain any words in the query

#11

#12

#13

#14

#15

#16

#17

#18

#19

#20

A) We pointed 20 links from the same pages/domains at this result with anchor text exactly matching

the query phrase

Page 98: Why Great Marketers Must Be Great Skeptics

#11

#12

#13

#14

#15

#16

#17

#18

#19

#20

#1

#2

#3

#4

#5

#6

#7

#8

#9

#10

After 16 days, all of the links had been indexed by Google. “A” moved up 19 positions to #1! B moved up 5 positions to #9. None of the other results moved more than 2 positions.

Page 99: Why Great Marketers Must Be Great Skeptics

Good thing we tested!

This is looking more conclusive, but we

should run at least one more test.

Anchor text = rankings. Stick a

fork in it!

Page 100: Why Great Marketers Must Be Great Skeptics

Does it influence Google’s non-personalized search rankings?

Google+

Page 101: Why Great Marketers Must Be Great Skeptics

Good discussion about Google+ correlations in this post

Google+ is just too damn high.

Page 102: Why Great Marketers Must Be Great Skeptics

Good discussion about Google+ correlations in this post

From a comment Matt Cutts left on the blog post:“Most of the initial discussion on this thread seemed to take from the blog post the idea that more Google +1s led to higher web ranking. I wanted to

preemptively tackle that perception.”

Page 103: Why Great Marketers Must Be Great Skeptics

Good discussion about Google+ correlations in this post

To me, that’s Google working really hard to NOT say “we don’t use any data from Google+ (directly or indirectly) at all in our ranking algorithms.” I would

be very surprised if they said that.

Page 104: Why Great Marketers Must Be Great Skeptics

Google explicitly SAID +1s don’t affect rankings. You

think they’d lie so blatantly? As if.

Page 105: Why Great Marketers Must Be Great Skeptics

The correlations are surprisingly high for something with no

connection. There have been several tests showing no result,

but if all it takes is a Google+ post, let’s do it!

Page 106: Why Great Marketers Must Be Great Skeptics

First, remember how hard it is to prove causality with a public test

like this. And second, don’t let anything but consistent, repeatable, provable results sway your opinion.

Page 107: Why Great Marketers Must Be Great Skeptics
Page 108: Why Great Marketers Must Be Great Skeptics

#21

#22

#23

#24

#25

#26

At 10:50am, the test URL ranked #26 in logged-out, non-

personalized, non-geo-biased, Google US results.

Page 109: Why Great Marketers Must Be Great Skeptics

42 minutes later, after ~30 shares, 40 +1s, and several other G+ accounts posting

the link, the target moved up to position #23

#21

#22

#23

#24

#25

#26

Page 110: Why Great Marketers Must Be Great Skeptics

#21

#22

#23

#24

#25

#26

48 hours later, after 100 shares of the post, 95 +1s, and tons of additional posts, the result was

back down to #25

Page 111: Why Great Marketers Must Be Great Skeptics

At least we proved one thing – the Google+ community is

awesome. Nearly 50 people shared the URL in their own

posts on G+!

Page 112: Why Great Marketers Must Be Great Skeptics

Many G+ users personalized results, however, were clearly

affected.

Page 113: Why Great Marketers Must Be Great Skeptics

#21

#22

#23

#24

#25

#26

#27

#28

#29

#30

Something very strange is happening in relation to the test URL in my personalized results,

though. It’s actually ranking LOWER than in non-personalized

results.

Page 114: Why Great Marketers Must Be Great Skeptics

Could Google be donking up the test?

Sadly, it’s impossible to know.

Page 115: Why Great Marketers Must Be Great Skeptics

GASP!!! The posts did move the result up, then someone

from Google must have seen it and is messing with

you!!!

Page 116: Why Great Marketers Must Be Great Skeptics

Sigh… It’s possible that Jenny’s right, but impossible to prove. We don’t know for sure what caused the initial movement, nor can we

say what’s causing the weird personalized results.

Page 117: Why Great Marketers Must Be Great Skeptics

More testing is needed, but how you do it without any potential

monkey wrenches is going to be a big challenge.

That said, remember this:

Page 118: Why Great Marketers Must Be Great Skeptics

Phew! We’re not alone.

Via Chartbeat

Page 119: Why Great Marketers Must Be Great Skeptics

If I were Google, I wouldn’t use Google+ activity by itself to rank anything, but I would connect G+

to my other data sources and potentially increase a page’s

rankings if many pieces of data told a story of engagement &

value for visitors.

Page 120: Why Great Marketers Must Be Great Skeptics

Ready to Be Your Own Skeptic?