View
145
Download
0
Embed Size (px)
DESCRIPTION
No e-commerce site is perfect, and there will always be pressure to improve conversions and AOV. A/B and multivariate testing can be an incredibly effective tool when used strategically. Where do you start? What tools do you need? How can you get buy-in from all brand stakeholders? We'll give you the rundown.
Citation preview
PRESENTED BY:
ERIC HANSER – DIGITAL OPERATIVE
Culture of OptimizationEcommerce A/B and Multivariate Testing Primer
digitaloperative.com Tweet us @digitaloperativ
digitaloperative.com Tweet us @digitaloperativ
THESE ARTICLES WILL NOTGIVE YOU A STRATEGY
Some of these tests may be effective, and some may not. Probably more of the latter.
Every site is unique and has unique users with unique problems.
Without more concrete hypotheses, a failed test leads to a dead end.
digitaloperative.com Tweet us @digitaloperativ
NUMBER 1Getting your site into shape is just like getting yourself into shape.
There is no magic pill or simple solution. It is a lifestyle change.
digitaloperative.com Tweet us @digitaloperativ
OK. SO WHERE DO YOU START?
digitaloperative.com Tweet us @digitaloperativ
LET’S TAKE ONE STEP BACK.
WHAT DO WE MEAN BY “TESTING”
digitaloperative.com Tweet us @digitaloperativ
SPLIT TESTING: SIMPLE IDEA, LOTS OF POSSIBILITIES
A split test is a controlled, randomized experiment run on your live site.
Users to your site are split up and shown different versions of the site.
The users’ progress is tracked to determine which version will be most successful.
digitaloperative.com Tweet us @digitaloperativ
SPLIT TESTING: PLATFORMS
digitaloperative.com Tweet us @digitaloperativ
CRO = CONVERSION RATE OPTIMIZATION
digitaloperative.com Tweet us @digitaloperativ
WHAT HAPPENS IF YOU IMPROVE CONVERSION RATE,
BUT AOV GOES DOWN?
digitaloperative.com Tweet us @digitaloperativ
THE TERM CRO CAN BE SOMEWHAT MISLEADING
A “conversion” technically refers to a a user completed any desired action.
In ecommerce, the word “conversion” is almost always used in reference to making a purchase.
digitaloperative.com Tweet us @digitaloperativ
THE TERM CRO CAN BE SOMEWHAT MISLEADING
Split testing can be used effectively to help with everything from branding to design to marketing, etc.
By using the term “CRO,” some departments or colleagues might not see immediately that these tools and processes are relevant and helpful to the work they do.
digitaloperative.com Tweet us @digitaloperativ
NUMBER 2In building a culture of optimization, it may be helpful to limit how often you use the term “CRO.”
digitaloperative.com Tweet us @digitaloperativ
TO AVOID CONFUSION, USE THE TERM
OPTIMIZATION & TESTING
digitaloperative.com Tweet us @digitaloperativ
WHAT ARE THE KINDS OF TESTS WE’RE TALKING ABOUT?
digitaloperative.com Tweet us @digitaloperativ
A/B TESTS: SIMPLEST (AND RAREST)
A/B Testing simply means testing one element on a page. Each option is called a version or a variation.
Not limited to 2 variations. Sometimes called A/B/C or A/B/N testing.
Only a single element is tested: color, placement, wording, picture, etc.
If more than one of these elements change, it is no longer an A/B test.
digitaloperative.com Tweet us @digitaloperativ
TYPES OF TESTS: A/B TESTING
digitaloperative.com Tweet us @digitaloperativ
MULTIVARIATE TESTING: A BIT MORE TIME
When more than one element on the page is being tested.
Each possible combination of elements should be tested.
With more variations, it can take longer to get statistically significant results (more on this later).
All variations are still on a single page.
digitaloperative.com Tweet us @digitaloperativ
TYPES OF TESTS: MULTIVARIATE TESTING
digitaloperative.com Tweet us @digitaloperativ
MULTI-PAGE TESTING: USERS SEE MORE THAN ONE PAGE
If an element appears on more than one page, you must test the change on each page.
Also often called “Flow Testing.”
Multi-page testing may be either A/B or Multivariate.
digitaloperative.com Tweet us @digitaloperativ
TYPES OF TESTS: MULTI-PAGE TESTING
digitaloperative.com Tweet us @digitaloperativ
SPLIT TESTING TYPES
A/B Tests – A single-page test with a single element changed
Multivariate Tests – A single-page test with more than one element changed.
Multi-page Tests – A test that shows a change across more than one page in the user flow.
digitaloperative.com Tweet us @digitaloperativ
ALL OF THESE TEST RANDOMLY SPLIT USERS INTO GROUPS.
digitaloperative.com Tweet us @digitaloperativ
TARGETING: ALL USERS ARE NOT THE SAME.
Optimizing your site for all users is important, but speaking directly to specific segments is even more effective.
Many of the same tools that are used to run split testing can also be used to target specific audiences and optimize the site for them.
digitaloperative.com Tweet us @digitaloperativ
NOT ALWAYS A TEST:SEGMENTING AND
TARGETING
digitaloperative.com Tweet us @digitaloperativ
NUMBER 3Results from testing can lead to targeting.
When looking at results of your test, you may find that certain segments should be targeted differently.
digitaloperative.com Tweet us @digitaloperativ
SO, WHERE DO YOU BEGIN?
digitaloperative.com Tweet us @digitaloperativ
IT IS TEMPTING TO JUMP RIGHT IN
It may seem that there are things “wrong” with your site or that you can make tests based on “best practice.”
Tests have “winners” and “losers” - there will be scrutiny.
The more information you have, the better chance you have of making wins early on.
digitaloperative.com Tweet us @digitaloperativ
NUMBER 4Don’t start with testing!
If your first few tests are flops, confidence in the whole idea of split testing could be compromised.
digitaloperative.com Tweet us @digitaloperativ
START WITH LEARNING THE “WHATS” AND “WHYS” OF
YOUR SITE.
digitaloperative.com Tweet us @digitaloperativ
FIND THE “WHAT”DATA, DATA, DATA!
digitaloperative.com Tweet us @digitaloperativ
FIND THE “WHAT”
What are users doing or not doing? What do they like and/or dislike?
Learn all you can from analytics. Proper set up and tracking could take time.
Ask questions like:What are the biggest drop-off and exit points?What pages are most successful?What are the most optimal site flows? What are
the least optimal?
digitaloperative.com Tweet us @digitaloperativ
FIND THE “WHY”USABILITY TESTING
digitaloperative.com Tweet us @digitaloperativ
FIND THE “WHY”
With data collected from analytics, develop paths or a series of tasks for users to complete in a usability test.
Several tools are available to either conduct in-person or remote usability testing. Usertesting.com is one of the most popular.
Often, unexpected and surprising results surface that you would have not necessarily seen in the analytics alone.
digitaloperative.com Tweet us @digitaloperativ
YOU KNOW THE PROBLEMS.
NOW, MAKE A HYPOTHESIS.
digitaloperative.com Tweet us @digitaloperativ
START WITH THE HYPOTHESIS – THEN DEVELOP THE TEST
When making a test, base your hypothesis on the problem.
“If I change X, then Y should improve.”
Never make a test and THEN ask the question, “how will I know which variation won?”
Simply looking at overall ecommerce conversion will likely be inconclusive.
digitaloperative.com Tweet us @digitaloperativ
WHAT & WHY30% of traffic direct to product pages
63% bounce rate
“Best Practice” recommendation to increase the add-to-cart button size and the free shipping message
digitaloperative.com Tweet us @digitaloperativ
WHAT & WHYUsers had no validation that the product was right for their specific condition.
Users had no clear way to find the right brace for them.
digitaloperative.com Tweet us @digitaloperativ
GETTING RESULTSSTATISTICS, BLEH!
digitaloperative.com Tweet us @digitaloperativ
GETTING RESULTS – DETERMINING THE WINNER!
Two primary metrics:Percentage Increase = How much a variation
improved over originalConfidence Interval = How confident we are
that the improvement is not due to chance
A 10% improvement with an 85% confidence interval basically means that there is an 85% chance that the test will result in a 10% improvement.
95% confidence interval = Statistically Significant
digitaloperative.com Tweet us @digitaloperativ
GETTING THE RESULTSSTATISTICAL SIGNIFICANCE
10% Improvement85% Confidence Interval
10% Improvement95% Confidence Interval
15% Improvement75% Confidence Interval
Original
digitaloperative.com Tweet us @digitaloperativ
WIN, LOSE, OR DRAW
Optimization is ongoing and iterative.
Ultimately, if a test is based on data and usability, we’re always learning and improving.
digitaloperative.com Tweet us @digitaloperativ
NUMBER 5If your testing is always based on data, every test will be a win.
If your tests aren’t based on data, even wins may be losses.
digitaloperative.com Tweet us @digitaloperativ
WINS & LOSSES – WHEN A WIN IS NOT A WIN.
If a test is implemented without a clear hypothesis and clear goals, a win can lead to a dead end.
If you don’t have a clear idea as to WHY a variation won, we’ve got nowhere to go.
digitaloperative.com Tweet us @digitaloperativ
WHEN A WIN IS NOT A WIN
digitaloperative.com Tweet us @digitaloperativ
WINS AND LOSSES – WHEN A LOSS IS NOT A LOSS
Similarly, we can learn a lot when a variation underperforms if the hypothesis was based on data.
With new information in mind, we can look again at the problem with additional data to find other solutions.
digitaloperative.com Tweet us @digitaloperativ
digitaloperative.com
WHEN A LOSS IS NOT A LOSS
digitaloperative.com
digitaloperative.com
WHEN A LOSS IS NOT A LOSS
WINS & LOSSES – WHEN A DRAW IS A WIN
Occasionally, a test goal will be something less “tangible” than an increase in conversions.
Some tests’ goals, for example, will be to bring consistency to a cluttered site, make copy more on brand, or to adapt to changing business requirements.
digitaloperative.com Tweet us @digitaloperativ
WINS AND LOSSESWHEN A DRAW IS A WIN
digitaloperative.com Tweet us @digitaloperativ
MAKE A PLAN – PRIORITIZE!
Not everything can be tested at once.
The P.I.E prioritization process helps put all possible tests in a digestible order.
P = Potential / How big is the problem?I = Importance / How valuable is the traffic?E = Ease / How difficult will the test be to
implement?
digitaloperative.com Tweet us @digitaloperativ
MAKE A PLANPRIORITIZE!
WEBSITE TESTING PrioritizationTest Name HYPOTHESIS Potential Importance Ease PIE AVG PIE Notes
Product Page - Short Product Description
By giving users feedback related to their path (filters, categories, search terms) they will be more confident in purchasing 8 9 7 8
High Traffic, High Bounce Rate, text-only implementation, but need use of audiences tool
Cart / Global Nav - View Cart
Design and placement of "view cart" CTA will bring more people back into the purchasing funnel if they leave the cart 6 8 9 7.666666667
Valuable traffic, but most users are directed to the shopping cart. Easy implementation
Category - Subcategory/Filter Design
By providing clarity about the differentiation between subcateory links and filters, potential UX issues will be mitigated 8 8 7 7.666666667
Noted problem/issue in usability. Use of filters shows intent, and therefor important.
Checkout - Security Messaging
Overstating Security may actually be harming conversion. Possibly finding a more optimal amount of messaging could improve it 5 9 9 7.666666667
No known issue, but hightly valuable traffic, and extremely easy test to implement.
Product Page - Find Other Products
Allowing users on a PDP to easily click off to other similar products if it not appropriate for their needs, we decrease bounce rates 8 7 7 7.333333333
Lots of traffic, high bounce rate, but the feature would be new, and therefore importancei is measured
Global Nav - Elements and Design
Users perform better when accessing injury categories as opposed to body type cateogirs 8 8 6 7.333333333
As a global element, importance and potential is high. Implementation will be slightly more than others.
digitaloperative.com Tweet us @digitaloperativ
NUMBER 6Prioritization not only keeps you organized, it helps your organization understand the breadth and importance of optimization and testing overall.
digitaloperative.com Tweet us @digitaloperativ
CULTURE OF OPTIMIZATIONWEBSITE FITNESS TIPS
There is no magic pill – It’s a lifestyle change.
The term “CRO” can be confusing – Use “Optimization & Testing.”
Test then target – Some segments may need to be targeted differently.
Don’t start with testing – Gather data (“whats” and “whys”) first.
Base tests on solid data and hypotheses – All tests can be wins.
Stay organized – Keep a clearly prioritized list of possible tests.
digitaloperative.com Tweet us @digitaloperativ
QUESTIONS?
WWW.DIGITALOPERATIVE.COM
digitaloperative.com Tweet us @digitaloperativ
PLEASE USE THE OR APP TO RATE THIS SESSION
Thank you for attending!