Informed & AgileTest Driven Design

with Jon Innes

Quick Housekeeping

• Control panel on the side of your screen if you have any comments during the presentation

• Time at the end for Q&A

• Today’s webinar will be recorded for future viewing

• All attendees will receive a copy of the slides/recording

• Continue the discussion using #uzwebinar

Meet the Speakers

Jon InnesFounding PrincipalUX Innovation LLC

Robin RichardsonVP of MarketingUserZoom

4UX INNOVATION LLC

Overview

• The often misunderstood MVP• Agile vs. Lean UX—what’s the difference?• User data, the missing ingredient in Agile• Test-driven design—data beats opinions• Practical tactics and lessons learned• Q&A

#UZWEBINAR

5UX INNOVATION LLC

Crappy Product vs.

Minimum Viable Product

What’s the difference?

#UZWEBINAR

6UX INNOVATION LLC

The often misunderstood MVP

#UZWEBINAR

7UX INNOVATION LLC

Well executed MVP—Tesla Roadster

Tests key hypothesis and delights some customers

#UZWEBINAR

8UX INNOVATION LLC

The Minimum Viable Prototype

Tests key hypothesis and doesn’t require building a product

#UZWEBINAR

9UX INNOVATION LLC

MVP—It’s also contextual…

1984 2013

Yesterday’s MVP may not be viable in a changing market

#UZWEBINAR

10UX INNOVATION LLC

Find a big unmet need we can solve well…

Scott CookFounder Intuit

Focus on what you are trying to learn

#UZWEBINAR

11UX INNOVATION LLC

Agile vs.

Lean UX

What’s the difference?

#UZWEBINAR

12UX INNOVATION LLC

What is Agile?

Best practices for developing software including:• Holding regular short status meetings in a standup format• Requirements in the form of user stories or use cases• Defining tests for code before writing the code itself • Working in short iterations to create incremental improvements

Common variations include Scrum & XP and all have a mindset that is best described as “lean”

#UZWEBINAR

13UX INNOVATION LLC

What is Scrum?

Product Backlog

Sprint Backlog

Potentially Shippable Product

Increment

Daily Scrum Meeting

24 hrs

2-4 weekSprint

Image from: http://www.mountaingoatsoftware.com/

?

#UZWEBINAR

14UX INNOVATION LLC

What is Lean?

Taiichi Ohno’s set of values and practices for removing three types of inefficiencies:• 無駄 = muda wasted outputs• 斑 = mura unnecessary inconsistencies• 無理 = muri wasted efforts

It worked for Toyota…and it applies to other fields…

#UZWEBINAR

15UX INNOVATION LLC

The Waste?

Building what nobody wants…

#UZWEBINAR

16UX INNOVATION LLC

The Lean Startup way

http://theleanstartup.com/

Do you have a product?

Build & measure it!

#UZWEBINAR

17UX INNOVATION LLC

What is Lean UX?

As response to the overall trend of using agile and lean methods, Lean UX applies the lean principles to UX:• Sizing UX deliverables to fit agile timelines or “just in time”• Eliminating unnecessary internal deliverables • Reducing variability in UX deliverables that create waste• Collaborating more closely with non-UX team members• Getting user feedback earlier and on a more regular basis

For more on Lean UX see:http://www.slideshare.net/balancedteam/02-sat-janice-lean-ux-landscapehttp://joshuaseiden.com/blog/2011/09/what-makes-it-lean/ http://uxdesign.smashingmagazine.com/2011/03/07/lean-ux-getting-out-of-the-deliverables-business/

#UZWEBINAR

18UX INNOVATION LLC

The Lean UX way

http://www.slideshare.net/clevergirl /

Do you need to build a product?

Prototype & measure it!

#UZWEBINAR

19UX INNOVATION LLC

Is the MVP Done?

Data beats opinions

#UZWEBINAR

20UX INNOVATION LLC

Not all ideas are bright…

http://en.wikipedia.org/wiki/File:Carbonfilament.jpg

#UZWEBINAR

21UX INNOVATION LLC

The key? Iterate and measure as you go...

Take your best shot, assess & try again

#UZWEBINAR

22UX INNOVATION LLC

Good metrics aligned to clear goals

Define objective metrics as team goals

#UZWEBINAR

23UX INNOVATION LLC

• Ship a desirable car with a 200 mile range

• Get users to add 7 friends in 10 days

https://www.teslamotors.com/blog/miles-progresshttp://genius.com/Chamath-palihapitiya-how-we-put-facebook-on-the-path-to-1-billion-users-annotated

Clear Goals

Create Great Products

#UZWEBINAR

24UX INNOVATION LLC

Dave McClure’s Categories of Metrics

• Marketing oriented, get user to know offering exists

• Get user to engage with product for initial 1st use

• Get user to integrate offering into their lives and use long term

• Get user to recommend to a friend

AARR!These cover the entire product lifecycle or UX

Categories based on Dave McClure’s AARRR Startup Metrics for Pirates see: http://500hats.typepad.com/500blogs/2007/06/internet-market.html

Acquire Activate Retain Refer

#UZWEBINAR

25UX INNOVATION LLC

Collaboration Via Shared Goals

http://flickr.com/photos/15489034@N00/420003710

#UZWEBINAR

26UX INNOVATION LLC

Making Progress?

Candidates for design metrics to guide teams:• # of interactions with target user segments (personas)• Sign-up rates• Retention rates• Task completion rates• Time on task• Error rates• Satisfaction scores• Net Promoter Scores

Focus on testing the right things

Be thoughtful about what you measure!

#UZWEBINAR

27UX INNOVATION LLC

Drive hypothesis and goal setting• Questions to ask on your team

• Have we validated these stories and personas? • Did the iteration incorporate measurable user feedback?

• Did you increase usage?• Do users they like it?• Can they use it?• Would they recommend it to a friend?• Did we make it measurably better?

• UX principles remain the same in Agile• Iterations without user tests are a lost learning opportunity• Waiting until the end test with users is the waterfall way• Focus on measurable user behaviors not team’s opinions• “Good enough” or “done” should be determined by users

#UZWEBINAR

28UX INNOVATION LLC

Do customer validation interviews

http://steveblank.com/

Do you have a market/user?

Get out of the building!

#UZWEBINAR

29UX INNOVATION LLC

Brief Description Married with 2 young children, Katie is a working mom who wants to cook more at home. She enjoys sharing recipes with friends and considers herself a “foodie”.

Computer Gear •Apple MacBook Air (2013)•iPhone 6s•iPad (3rd gen)

Pain Points Wants recipes that are nutritious and healthy but quick to make, not recipes that are incomplete or hard to follow. No ingredients she can’t find easily. Hates spending lots of money on exotic ingredients only to find her kids hate the taste. Avoid any extra trips to the store during her busy week for items she forgot, or can’t find easily. Dealing with other children’s food allergies.

Values Being considered a good cook. Likes organic and healthy foods but willing to compromise for dishes that are really tasty or that her kids or spouse really love. Simplicity over complexity, but willing to put a little extra effort in to make things that taste great or are really healthy. Saving money by cooking food at home.

Key User Stories US1 Find recipes for ingredients I have on hand to avoid extra shopping tripsUS2 Create a weekly shopping list with ingredients for recipes before shoppingUS3 See recipes my friends recommend for ideas to plan mealsUS4 See recipes famous chefs recommend for ideas to plan meals US5 Save recipes with cooking notes so I can reuse them when cooking mealsUS6 Share my recipes and cooking notes with my friends and family US7 Find recipes for things on sale when planning meals to save money

None yet

Related User Research

Average Task Completion Rate = ?%System Usability Score = ?

Net Promoter Score = ?Experience Rating = ?

Add stories and metrics to personas

#UZWEBINAR

30UX INNOVATION LLC

Use story maps to identify MVP stories

See Jeff Patten’s articlewww.agileproductdesign.com/writing/how_you_slice_it.pdf

#UZWEBINAR

31UX INNOVATION LLC

Iterating without user feedback is fast but futile…

Sprint towards clear goals

Measure progress

#UZWEBINAR

32UX INNOVATION LLC

Who?1

What? 2

Groom 3

Estimate 4

Assign5

Market Metric 7

Story Metric 6

Bonus:Add hyperlinks to deliverables

#UZWEBINAR

33UX INNOVATION LLC

The Challenge: Testing in an Agile Way

• Traditional testing does not fit well into agile:• Recruiting can take up to two weeks• Reports and test plans take too long• You often don’t have a stable UI until too late

• What can we do? Adapt existing methods?• Include design research tasks as part of sprint planning• 3x3 solves the “waiting till the end” for working UI problem• RITE is a step in the right direction, it speeds up iterations• A/B split tests are effective for tweaking final details• Automate with RUT when at all feasible

#UZWEBINAR

34UX INNOVATION LLC

Design Refine

Combine

The 3x3 way: Test multiple concepts early

http://www.carolrighi.com/documents/Righi%203x3.pdf

• 3 concepts, 3 pages deep

• 3 iterations of design & test

• 3 or more users per iteration

• Test low fidelity first, then high

• Improve concept each iteration

• Combine best elements into 1 concept & iterate

Refine Design

#UZWEBINAR

35UX INNOVATION LLC

The RITE way: Refine one concept faster

Schedule image from Brian Keith Sullivanhttp://bigdesignevents.com/2011/08/5-ways-to-improve-your-next-agile-project-with-the-rite-method/RITE: Method http://en.wikipedia.org/wiki/RITE_Method

REFINEDESIGN

REFINEDESIGN

REFINEDESIGN

#UZWEBINAR

36UX INNOVATION LLC

The A/B way: Build and compare

Design A Build & Launch A Test A

Design B Build & Launch B Test B

• Design & build multiple designs and compare details• By nature less subject to test condition biases• Easy to get large sample sizes & statistically reliable metrics• Not so great at providing deeper insights on “why”

Update

Repeat

#UZWEBINAR

37UX INNOVATION LLC

Remote Automated Usability Testing (RUT)

• Not just site analytics or surveys• Task oriented vs. page views and time on site• Not limited to existing users of your site• Can be used on prototypes or working code• No moderator required—scales with low effort• Combines the benefits of lab testing with the

scalability of A/B testing

#UZWEBINAR

38UX INNOVATION LLC

Remote Automated Testing supports

• Task completion rates vs. click counts• Time on tasks vs. time on site• Click path data by user task• Heat maps showing where users click

on a page• Task specific and overall comments• Survey data (e.g., SUS or NPS )• Video highlights • Larger sample sizes

#UZWEBINAR

UserZoom at a Glance

• All-in-one solution to rapidly test Usability & Measure UX

• Remote Unmoderated Usability Test (Qualitative or Think-out-loud)

• Usability Benchmarking • Card Sorting• Tree Testing• Screenshot Click Testing• Screenshot Timeout Testing• Online Surveys

• Test any digital UI at anytime during the product life cycle

The All-in-One UX Research Platform for the Data-Driven Enterprise

40UX INNOVATION LLC

Tips for Remote Automated Testing

• Always run small pilots before large scale tests, in many cases this can provide valid data anyway

• Allow at least a week for tests with panel participants• Combine intercept recruiting with panels to minimize

costs while reaching both new and existing users• Keep the participant tasks to a small number to

minimize drop out rates• Use content analysis techniques to analyze open

ended survey data

#UZWEBINAR

41UX INNOVATION LLC

More Agile tips

• Have personas agreed to by team before sprinting• Develop an appropriate recruiting strategy early• Define hypotheses & target metrics for stories in the

backlog working with your product owner• Track usability metrics in a public place• Test early design concepts not just final details• Automate UX testing for stories as you go & retest• Waiting until the end is the waterfall way

#UZWEBINAR

42UX INNOVATION LLC

It is not the strongest of the species that survives, nor the most intelligent. It is the one most adaptable to change. Charles Darwin

Final thoughts

#UZWEBINAR

43UX INNOVATION LLC

Q&A

#UZWEBINAR

44

Need more info?Get answers about UserZoom. We are here to help. Talk with one of our experts.

Contact us

UserZoom10 Almaden Blvd. #250San Jose, CA 95113

Toll free number: (866) 599 1550

www.linkedin.com/company/userzoom@UserZoomFollow us