The Science of Great Site Navigation: Online Card Sorting + Tree Testing

Jeff Sauro | Measuring Usability LLC

@userzoom #uzwebinar

CARDS AS PSYCHOLOGICAL APPARATUS

Traditional Card Sort

Online Card Sorting

Card Sorting is as common as Lab based Usability Testing

Source: 2011 UxPA Salary Survey

Terms & Concepts

• Open Sort: Users sort items into groups and give the groups a name.

Closed Sort: Users sort items into previously defined category names.

• Reverse Card Sort (Tree Test) : Users are asked to locate items in a hierarchy (no design)

• Most Users Start Browsing vs Searching: Across 9 websites and 25 tasks we found on average 86% start browsing

http://www.measuringusability.com/blog/card-sorting.phphttp://www.measuringusability.com/blog/search-browse.php

Target.com Case Study

Method & FAQs

• We want to improve the Target.com navigation. – Obtain a baseline set of metrics and problem spots– Make improvements– Compare against baseline

• Pick a variety of items to test the taxonomy• Conduct an Open Sort then Tree-Test• Use Product Names with some level of context

40 Items Across Departments1. Presto GranPappy Electric Deep Fryer2. Black CleanAirBall Air Purifier3. Nano Tech Shaver4. Hamilton Beach FashionFirst Medium Iron5. Brother International Sewing Machine-

XR9500PRW6. T3 White Micro Featherweight Hair Dryer7. Signature Sleep Silhouette 10” Queen8. Thomas O’Brien Canterbury Blue Bath Rug9. Fieldcrest Luxury Towel Collection10. Privet House Sugar Sack Pillow11. P Dog ZIP Lead Retractable Blue Leash12. Kitchen Aid 3pc BBQ Set13. Women’s Mossimo Kasey Boot14. Women’s Mossimo Pearce Black Leather Pumps15. Men’s Brown Swiss Gear Jackson Hiker16. Totes Clear Bubble Umbrella17. Merona Yellow Scarf Tote18. Wildkin Kaleidoscope Backpack19. NBA Golden State Warriors Blue Water Bottle20. Grey Stripe Merona Men’s Short-Sleeve Polo

21. Corellian Freighter Men’s Graphic Tee22. Inset Mother of Pearl Stainless Steel Cuff Links23. Xhilaration Black Large Sunglasses24. Wrangler Mens Reversible Belt25. Cherokee Boys Drawstring Cargo Short26. Girls Danshuz Pink Short-Sleeve Sparkle Straight

Dress27. Tiddliwinks Blue Dot Plush Monkey28. Black Griffin Case for Apple iPod touch29. Marvel POWER ATTACK AVENGER: IRONMAN30. Titanic (2 Discs) (Blu-ray/DVD)31. PC Diablo 3 Computer Game32. PC Treasures FlipIt 10” Tablet Sleeve33. Commodore Big And Tall Executive Oversize Chair34. Classic Borders Wedding Invitation and Response

Card35. Hello Kitty Party Favors 32-pc.36. Women’s Obsession by Calvin Klein Eau de

Toilette37. C9 by Champion Women’s Molded Cup Racer

Back Sports Bra38. Coffee Shop Outerwear Women’s Plus-Size Spring

Twill Coat39. Women’s One Shoulder Shantung Dress40. $5-$1,000 Iconic Puppy GiftCard

Overview

• 50 Participants– 65% Female– 2 Excluded for Quality Control (Haphazard Sorts)

• Sort 40 Items into 10 Fixed Categories• Median Study Time: 11 Minutes• Output– Dendrogram of Clusters– Open Category Names– Most Difficult to Sort

12 Clusters

1. Men’s / Boys2. Women’s / Girls3. Accessories 4. Runt: NBA5. Kitchenware6. Bed + Bath7. Runt : Furniture8. Toys9. Electronics10. Runt: Shaver11. Misc/Pets: Gift Card & Leash12. Misc: Party Favors & Invites

Dendrogram

Most Popular Category Labels

Similar Category Names CombinedPercentages are % of 48 participants who labeled a category, so % adds up to more than 100%Error Bars are 90% Confidence Intervals

Most Difficult Items to Categorize

Percentages are % of 48 participants who selected item, so % add up to more than 100. Error Bars are 90% Confidence Intervals

Dendrogram Close Up

Item 1st % 2nd % 3rd % 4th % 5th %Black Griffin Case for Apple iPod touch Electronics 65 Entertainment 15 Accessories 10 Misc 4 Gifts 2

PC Diablo 3 Computer Game Electronics 63 Entertainment 29 Misc 4 Other 2 Gifts 2PC Treasures FlipIt 10”Tablet Sleeve Electronics 58 Entertainment 13 Misc 10 Accessories 10 Gifts 2Grey Stripe Merona Men’s Short-Sleeve Polo Men 56 Clothing 40 Boys 2 Accessories 2Corellian Freighter Men’s Graphic Tee Men 56 Clothing 38 Sporting Goods 2 Accessories 2 Boys 2C9 by Champion Women’s Molded Cup Racer Back Sports Bra Womens 54 Clothing 38 Girls 4 Sporting Goods 2 Accessories 2Coffee Shop Outerwear Womens Plus-Size Spring Twill Coat Womens 54 Clothing 38 Girls 4 Homeware 2 Accessories 2

Black CleanAirBall Air Purifier Homeware 52 Electronics 13 Misc 8 Appliances 8 Bed & bath 2Women’s Mossimo Kasey Boot Womens 52 Clothing 27 Shoes 15 Girls 4 Accessories 2Women’s Mossimo Pearce Black Leather Pumps Womens 52 Clothing 27 Shoes 13 Accessories 4 Girls 4Women’s One Shoulder Shantung Dress Womens 52 Clothing 40 Girls 4 retail 2 Accessories 2P Dog ZIP Lead Retractable Blue Leash Pets 50 Misc 15 Accessories 10 Outdoor 8 Homeware 6Titanic (2 Discs) (Blu-ray/DVD) Electronics 50 Entertainment

40 Misc 4 Womens 2 Gifts 2

Men’s Brown Swiss Gear Jackson Hiker Men 48 Clothing 35 Accessories 6 Shoes 4 Outdoor 4Hamilton Beach FashionFirst Medium Iron Homeware 46 Electronics 15 Appliances 10 Misc 8 Womens 6Brother International Sewing Machine-XR9500PRW Homeware 46 Electronics 19 Appliances 10 Misc 10 Womens 4Xhilaration Black Large Sunglasses Accessories 46 Womens 25 Clothing 10 Men 6 Misc 6Wrangler Mens Reversible Belt Men 46 Accessories 29 Clothing 19 Womens 2 Boys 2

Totes Clear Bubble Umbrella Accessories 42 Homeware 15 Womens 13 Misc 10 Outdoor 4

Items with > 40% of Votes as Primary Category

Item 1st % 2nd % 3rd % 4th % 5th %Signature Sleep Silhouette Queen Homeware 40 Bedroom 17 Bed & Bath 10 Furniture 10 Accessories 4

Privet House Sugar Sack Pillow Homeware 40 Bedroom 17 Bed & Bath 13 Furniture 6 Misc 4Cherokee Boys Drawstring Cargo Short Clothing 40 Men 31 Kids 17 Boys 8 Homeware 2Girls Danshuz Pink Short-Sleeve Sparkle Straight Dress Clothing 40 Womens 35 Kids 15 Girls 8 Accessories 2

Nano Tech Shaver Electronics 38 Bath 21 Homeware 13 Men 10 Misc 4

Thomas O’Brien Canterbury Blue Bath Rug Homeware 38 Bath 25 Bed & Bath 13 Misc 6 Bed bath 4

Kitchen Aid 3pc BBQ Set Homeware 38 Kitchen 33 Appliances 10 Outdoor 4 Misc 4Women’s Obsession by Calvin Klein Eau de Toilette Womens 38 Bath 17 fragrance 10 Accessories 10 Clothing 6$5-$1,000 Iconic Puppy GiftCard Pets 38 Misc 15 Gifts 13 Cards/Party 8 Accessories 4Presto Gran Pappy Electric Deep Fryer Homeware 35 Kitchen 31 Electronics 13 Appliances 10 Misc 6

Merona Yellow Scarf Tote Womens 35 Accessories 29 Clothing 21 Homeware 6 Luggage 4Commodore Big And Tall Executive Oversize Chair Homeware 35 Furniture 31 Misc 10 Office 6 Other 4T3 White Micro Featherweight Hair Dryer Homeware 33 Bath 17 Electronics 15 Misc 10 Appliances 6Fieldcrest Luxury Towel Collection Homeware 33 Bath 31 Bed & Bath 13 Bedroom 4 Misc 4Inset Mother of Pearl Stainless Steel Cuff Links Accessories 33 Men 31 Womens 15 Clothing 13 Jewelry 6Classic Borders Wedding Invitation and Response Card Cards/Party 33 Misc 17 Accessories 13 Gifts 6 Womens 6Wildkin Kaleidoscope Backpack Accessories 31 Kids 19 Misc 10 Clothing 10 Outdoor 8

Hello Kitty Party Favors 32-pc. Cards/Party 29 Misc 17 Kids 15 Gifts 6 Accessories 6Marvel POWER ATTACK AVENGER: IRONMAN Entertainment 27 Toys 19 Electronics 19 Kids 13 Misc 10Tiddliwinks Blue Dot Plush Monkey Toys 23 Kids 23 Misc 19 Homeware 4 Accessories 4NBA Golden State Warriors Blue Water Bottle Misc 21 Accessories 21 Sporting Goods 13 Kitchen 10 Men 10

Items with < 40% of Votes as Primary Category

Tree Test

Selected 14 Items From Open Sort

1. Wrangler Mens Reversible Belt2. Marvel POWER ATTACK AVENGER: IRONMAN3. Women’s One Shoulder Shantung Dress4. Grey Stripe Merona Men’s Short-Sleeve Polo5. Inset Mother of Pearl Stainless Steel Cuff Links6. Women’s Obsession by Calvin Klein Eau de Toilette7. Tiddliwinks Blue Dot Plush Monkey8. Hello Kitty Party Favors 32-pc.9. Nano Tech Shaver10. Brother International Sewing Machine-XR9500PRW11. Classic Borders Wedding Invitation and Response Card12. Kitchen Aid 3pc BBQ Set13. Men’s Brown Swiss Gear Jackson Hiker14. Wildkin Kaleidoscope Backpack

Overview

• 50 Participants• 68% Female

• Median Study Time 17 Minutes• Task Based Metrics• Confidence• Difficulty (SEQ)• Success• Time

Post Task Questions in Userzoom

Single Ease Question: SEQ

Tree Testing in Userzoom (User’s View)

Successfully Located Item

Hardest Items to Locate

Brother Sewing Machine 64% -The sewing machine didn’t have a separate section that I could find- it could fit into a lot of categories and it made it hard to nail down

Wildkin Kalidescope Backpack 38% - Not sure if it’s a child’s toy or real camping gear- Expected to find it in the children’s section

Hello Kitty Party Favors 36% - I didn’t find a section for things like Cards or Party Favors Could be for both men and women- not apparent under what main heading it would be under.

Men’s Brown Swiss Gear Jackson Hiker 34% - Did not know what this was, a boot? A Jacket?

Classis Border Wedding Invitations 39% - Was expecting to find a stationary or cards section

Twiddledinks Blue Dot Plush Monkey 12% - Did not know if this was a toy- No Stuffed animal section

Kitchenaid 3-pc BBQ set 10% -Did not know if it was Kitchen or a small appliance- Could be categorized under kitchen or outdoors

Marvel Attack Avengers: Ironman 8% - Did not know what kind of Toy it was- List of Toys was too long, it was hard to eliminate

Nano Tech Shaver 6% - Is it for a man or a women?- Did not know if it was electronics or another category

Women’s Obsession by Calvin Klein 6% - Did not know which category to put it under

Mother of Pearl Stainless Steele Cuff links 2% - Could have fit into more than one category

Item % that found it difficult Reason it was difficult

Verbatim Reasons For Difficulty

How difficult or easy was it to complete the task

Very EasyVery Difficult

Percentages are normalized percentile ranks from across 200 tasks. 50% = Average.

Confidence

Not at all Confident

Extremely Confident

Confidence & Success

DisastersDifficult

Deceptive

Hardest Items to Categorize – Tree Test Vs Open Sort

Difficulty sorting explains about 16% of difficulty

finding.

Sample Size ConsiderationsMargin of Error

90% Confidence (+/-) Sample Size24% 1020% 1515% 2814% 3213% 3812% 4511% 5410% 659% 818% 1037% 1366% 1865% 2684% 4213% 749

See Chapter 6 in Quantifying the User Experience

Conclusions

• Determine if new categories should be created or consolidated

• Look to confirm or deny hypotheses about categories and findability– Gender Searching– Sub Categories

• Watch for “false” sorts• If “key” items are hard to find, move them,

test them• It’s hard to do a Card Sort without a Tree Test

Find out More

MeasuringU.comjeff@measuringu.com