Upload
quentin-christensen
View
59
Download
1
Embed Size (px)
DESCRIPTION
Guerilla HCI is low cost methods of learning from customers and testing your products to improve them. Learn about the different types of guerilla HCI methods you can use to build great products when you don't have unlimited resources to interact with customers and run expensive research studies.
Citation preview
Agenda• Customer Activities• Research Methods• Shipping Activities
Special thanks to Jakob Nielsen, his 1994 Guerilla HCI article is the basis for this presentation• http://en.wikipedia.org/wiki/Comparison_of_usability_evaluation_methods• http://www.useit.com/papers/guerrilla_hci.html• http://www.useit.com/alertbox/discount-usability.html
Cost savings in a medium scale software project
Original usability cost estimate by [Mantei and Teorey 1988] $128,330Scenario developed as paper mockup instead of on videotape - $2,160Prototyping done with free hypertext package - $16,000All user testing done with 3 subjects instead of 5 - $11,520Thinking aloud studies analyzed by taking notes instead of by video taping - $5,520
Special video laboratory not needed - $17,600Only 2 focus groups instead of 3 for market research - $2,000Only 1 focus group instead of 3 for accept analysis - $4,000Questionnaires only used in feedback phase, not after prototype testing - $7,200
Usability expert brought in for heuristic evaluation + $3,000Cost for "discount usability engineering" project $65,330
Source: Jakob Nielsen 1994 http://www.useit.com/papers/guerrilla_hci.html
Customer Activities Research Methods• Focus Groups• Customer Focused Design• Customer Visits• Councils
• Customer• Partner• Marketing/Sales
• Customer Buddies• Internal Partners• Partner Relationships
• Concept Box Testing• Usability Lab Testing• Discount Usability Testing
• Scenario• Customer design review• Simplified think aloud• Pluralistic walkthroughs• Heuristic evaluation
• UX/UI/String Reviews• Polish SWAT• Real world projects• Scenario testing & health• White papers• Early customer deployment• Dogfood
Shipping Activities
Customer Activities
• Focus Groups• Customer Focused Design• Customer Visits• Councils
• Customer• Partner• Marketing/Sales
• Customer Buddies• Internal Partners• Partner Relationships
What should I build?
Focus Groups / Customer Focused Design
• Focus Groups• Create a topic• Get several people in a room• Ask them questions• Record their responses
• Customer Focused Design• Same as focus group• List out the design ideas• Have the room rate the design ideas• Confirm the priorities with the group
Customer Visits• Travel to the customers location• Spend a day or two meeting various users• Use one on one interviews and focus groups• Try to collect artifacts if possible• Take detailed notes
Councils• 3-20 people that are experts in the area of the problem you
are solving• 3 Types:
• Customers – people that will use your product• Partners – developers that will extend your product• Marketing/Sales – the people that will sell your product
• Create the council when you begin planning the product• Meet with the council throughout development• Quarterly meetings
• Collect feedback• Present early designs Product demos• Discuss product plans• Show UI
Customer Buddies• 1-3 users that you have a deep ongoing relationships with• Frequent meetings and emails• Conduct UI walkthroughs during iteration, early & often• Discuss major design problems and design alternatives• In depth design discussion• Clarification of requirements
Internal Partners• People at your organization that will be “users” of your
product• Easy to setup a meeting and discuss issues!• Help find usability study participants• In depth discussions• UI walkthroughs• Dogfood preparation
Partner Relationships• Relationships with developers that will build on or extend
your product• Get their feedback as experts that understand the market• Design a platform that supports their needs• Help them go to market earlier and increase adoption• Two types:
• Independent Software Vendor (ISV) – sell and markets an app that uses your platform
• Systems Integrator (SI) – sells your app and helps customers deploy and integrate with other software
Research Methods
• Concept Box Testing• Usability Lab Testing• Discount Usability Testing
• Scenario• Customer design review• Simplified think aloud• Pluralistic walkthroughs• Heuristic evaluation
How should I build it?
Testing With 3-5 Users
http://www.useit.com/alertbox/20000319.htm
Scenario• Explain in a paragraph• Reduce to the bare minimum• Small and easy to change• Use low cost prototypes to
describe and test on users
Try this with:• Your team members• UX• Individual users
When:• You first start designing
Example: Jim is authoring a document on hurricane patterns in the Gulf of Mexico. He shares the document with Armando, who works at a different university, for review. Armando adds several comments and Jim is delightfully surprised that he can see the comments right in the document and easily make the updates.
Customer Design Review• Meet with a small group of users• Present a design / prototype• Explain the design and functionality• Lead the discussion• Ask probing questions• Focus on uncovering requirements
Try this with:• Individual users • Groups of customers
When:• You have a decent prototype• Early and late in the design
Simplified Think Aloud• Test one user at a time• Have a list of tasks / use cases• Show a prototype• Users perform the tasks• Detailed design, but low fidelity• Take notes and measure performance• Try testing multiple designs
Try this with:• Individual users• UX• Your team members
When:• Nearly final design• After coding + some polish
I have no idea
what to do next.
Cognitive / Pluralistic Walkthroughs• Walk through the design with people from various backgrounds
together • Ask everyone to write on paper how they would accomplish each task
Discuss amongst the group• Present the correct answer if the group goes too far off course• Don’t lead or try and explain everything to users• Have a list of tasks and a prototype
Try this with:• Customers• UX• Your team members
When:• Nearly final design• After coding + some polish
Heuristic Evaluation• 3-5 evaluators• Have a prototype or actual implementation• Walk through the design• Rate the design based on heuristic standards
Try this with:• Your team members• UX
When:• Any time
Heuristic Evaluation1. Visibility of system status The system should always keep users informed about what is going on, through
appropriate feedback within reasonable time. 2. Match between system and the real world The system should speak the users' language, with words,
phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
3. User control and freedom Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
4. Consistency and standards Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
5. Error prevention Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
6. Recognition rather than recall Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
7. Flexibility and efficiency of use Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
8. Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
9. Help users recognize, diagnose, and recover from errors Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
10. Help and documentation Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.
http://www.useit.com/papers/heuristic/heuristic_evaluation.html
Microsoft eDiscovery Examples• User tasks• Score sheet• Scenario Health• Design for eDiscovery Preservation
User Tasks1. A legal event has occurred that requires you to hold content
1. Identify the mailbox for Dan Jump and Quentin Christensen, document center SharePoint site, and file share
2. Get an estimate of how many items there are3. Reduce the amount of content using some keywords4. Hold the content to make sure it cannot be deleted5. Gather content to ensure it is held
2. You know more about the case and need to limit the content so you can export it, but you don’t want to affect how it is preserved
1. Use keywords to limit the data set2. Get the content to the correct set of data you need3. Close your work so you can access it another time
3. You now need to gather the data set you have specified 1. Select the content you want to gather2. Send it to a location where you can easily access it3. Verify you have the correct content and are complete
4. You need to print a report to prove you adequately held content 1. Create a report that shows the dates of all the content you preserved
Heuristic Score Sheet
http://office/15/teams/exchange/SEC/FindMyVault/FederatedFind/Shared Documents/eDiscovery Scenario Status.xlsx
Scenario HealthStatus Scenario Task Bugs/Notes
Create a case
Create a discovery set
Add a mailbox
Add a SharePoint site
Get the number of items
Enter a filter and view the stats
Preserve and save the discovery set
Check the preservation in Exchange Delete an email - notice it is kept in the dumpster Check the SharePoint site see it is preserved
Edit a doc - it is copied to the preservation library
Delete a doc - it is copied to the preservation library
View auditing report of when content was preserved Create a new query Specify a query with a wildcard and proximity
Narrow down the query scope
View results from Exchange, SharePoint, and file shares
Show the hovercard with a preview of the message Open the message Select multiple refinements for SharePoint
View the hovercard for SharePoint items
Save the query
Export the query
Design for eDiscovery Preservation
3:00PM1/23/2008
Search SaveEstimate Hits
eDiscovery
Document Name – Windows Internet Explorer
Favorites | Website 1 Website 2 Website 3
http://mysite/ediscovery Search
HomeCustodiansLocationsSearchesExportsReports
All of these words:The exact phrase:
Any of these words:
None of these words:
Scope
Date Range: to
Email Property Restrictions
File Property RestrictionsLanguage Restrictions
All Mailboxes
Mailboxes
All Locations
Locations
Result Type
Keywords Keywords:
Add Mailboxes Add Locations
Add Mailbox
CancelSave
Custodian: Lionel Penuchot
Add To Case Name: Luka AbrusPreserve MailboxNotify
Place Associated Locations on Hold
Associated Locations
><
Import Locations
Lukas DesktopFileshare BTeam SiteProject Team Site
3:00PM1/23/2008
eDiscovery
Document Name – Windows Internet Explorer
Favorites | Website 1 Website 2 Website 3
http://mysite/ediscovery Search
Case Manager Home
Cases
Holds
Custodians
Content Sources
Content Sources
Add Content Source
Name Type Items Size
Data Wiz Site SharePoint Site 3,460 1.384GB
Data Wiz Files
Fileshare 3,403 1.323GB
3:00PM1/23/2008
eDiscovery
Datum v. Contoso – Windows Internet Explorer
Favorites | Website 1 Website 2 Website 3
http://portals/sites/ediscovery/contoso Search
HomeIdentify & Preserve Custodians LocationsSearch and Process Searches ExportsReportsCase Settings
Case Owners
Status
Preservation Settings
Tom Perham;
Active
Preserve all content Select preservation type per
mailbox Filter preserved contentDate Range: toQuery:
Add Additional Filters
Mailboxes
Preserve all content Select preservation type per
location Filter preserved content
Locations
Preview
3:00PM1/23/2008
eDiscovery
Document Name – Windows Internet Explorer
Favorites | Website 1 Website 2 Website 3
http://mysite/ediscovery Search
Case Manager HomeCasesCollections
Create a New Case Datum v. ContosoName:
Tom PerhamCase Owner:
CancelSave
Collections
Add Collection
None
Collections
CancelSave
Name Date Created Managed By On Hold TypeWidget F Files 9/26/2010 Tom Perham Yes Mailbox; SharePoint
Steve Den 3/7/2010 Tom Perham Yes Mailbox
Widget D 2/4/2010 Tom Perham No SharePointCreative Files 6/2/2009 Luka Abrus Yes File Share;
SharePointController Files 3/6/2009 Tom Perham Yes Search Results
Active Collections | Closed Collections | Create New Collection
Add Collection
Filter:
Preview
Date Range: to
Preserve
Add Sources
Sources
Preservation Settings
Preserve this content
Save and Apply ChangesCancel
Update Statistics
Filter (optional)
Add a filter to reduce the content (e.g. date>10/10/2010, images)
Collection Statistics
Source Item Count
Size Cull Rate
Total 0 0 0%
Name: Name your collection
Preservation Type Statistics
Content Type Statistics
Add a collection
Queries
CASE HOMECOLLECTIONSSEARCHESEXPORTSSOURCESDOCUMENTSREPORTS
Filter (optional)
Add a filter to reduce the content (e.g. date>10/10/2010, images)
Name: Name your collection
Date: to:Sources
Name Source Type Quentin Christensen Exchange
Mailbox http://contoso/teams/sales SharePoint Site //back/scratch/quentin File Share
Add sources
Searches
Name Modified Items Size
Add new item
Preservation Type StatisticsContent Type Statistics
Statistics
Source Items Size
Total
Preservation Settings Preserve this content Collect content
Save CancelUpdate Statistics
Preview Results
HOMECOLLECTIONSQUERIESSOURCESEXPORTSDOCUMENTS
Apply Filter
Save CancelPreview Results
Name* Name your collection
Sources
Source Source Type Preservation Status Items Size
TotalManage sources
Filter
Add a keyword to filter content
Start Date: End Date:Keywords:
Preservation
Certain repositories such as SharePoint and Exchange support in place preservation. Selecting preserve this content will protect it in its original location.
Preserve
Author:
There are no sources in this collection. To add sources, click “Manage Sources”.
Initial Design
Initial Build
Redesign
Beta1
Shipping Activities• UX/UI/String Reviews• Polish SWAT• Real world projects• Scenario testing & health• White papers• Early customer deployment• Dogfood
UX/UI/String Reviews• Get team members, UX, design in a room• Walkthrough the experience• Testers write down bugs and file later
• UX/UI – focus on the overall experience, flow, interactions, look and feel. Get into the details on spacing, color, font, etc.
• String – Focus on the text that is displayed to users, be sure to cover all error cases• Get a list of all text in areas you own• Try to view it in context, but be sure to review everything• Get recommendations on how to change the string
Polish SWAT
No, this Polish SWAT1. Identify page/area/experience to polish2. File one bug to track the SWAT3. Identify existing bugs and track them4. Do a UI review, identify all issues5. Track issues in a shared OneNote6. Review with your team7. Create a design with all changes 8. Fix all issues9. Dev sends link to changes10. Review in person, identify remaining
issues, fix, repeat11. Your done:
1. Everyone agrees its ready2. Polished – no remaining UI bugs
Members:• Dev / Test• UX/Design• SMEs
Polish SWAT Tips• Double check CSS – make it consistent (IE F12 Developer
Tools)• Get it right - but don’t stress UI and CSS you don’t control, go
for consistency not the way you want it to look• Meet – with the developer, make real time changes• Focus – pick one page or task• Schedule a week – the SWAT is a top priority, meet ad hoc
not on a schedule • Agility – move to each step as fast as possible, if the
developer completes the update meet immediately Agreement – its done when you all agree its done
• Its Done – the page looks ready for customers (aside from any issues you don’t control)
Example: Microsoft eDiscoveryBefore SWAT
After SWAT23 initial issues identified10 new issues identified during iterationLooks way better after 1 week
Spec
Real World Projects• Build real world applications using the latest build• Build a real thing, use real data:• Team up – get 3-4 people• Dedicate a week – it should be the majority of your time for
that week• File as many bugs as possible! 50-100 for a team or even
one person is doable• Can you complete the scenario?
Example: Microsoft Sales Portal• Rebuild the Microsoft Sales Portal site on the new version
of the product (Alpha Build)• 1 week, 4 people full time,112 bugs filed
Scenario Testing & Health• Identify core tasks for your scenario• Conduct regular test passes• Rate each task
• Green – no issues or very minor issues• Yellow – minor visual issues, not great
experience• Red – major visual or functional issues• Black – cannot complete task
• Mark bad bugs scenario blocking• Track usability & experience issues
ExampleID Assigned To Sub Status Title290742 ramansom Paging doesnt work for exchange sources319499 boxinli Bug
UnderstoodWhen an update panel refreshes search no longer works
325512 carloa No stats on the saved searches list view; need Items and Size328714 srivr Auth: The realm in the token is hardcoded to 'microsoft.com'333406 shiyao Sum of # of documents displayed in refiners is greater than the total
number of documents returned in the query results (for very simple result sets)
350069 boxinli Need a way to cancel a query which has been issued to the SDP360106 carloa Case Manager: When failing to talk to search, the seaching window in the
query edit form keeps searching forever
361198 carloa Creating a new Query should search all sources in the case with a blank filter
373183 ramansom Search fail to get token when searching using OAuth376625 grahamc Local Fix Exchange SourceFilter includes Path property; Exchange exports don't
work
2295591
sveinar Local Fix Thread.CurrentPrincipal in the flow is not correct
2314397
carloa The hover panel is gone; Object required in searchui.js line 1 col 9342
2342688
carloa Refiners fall back to non Discovery refiners if you specify a specific collection as the query scope
2345525
carloa Loading advanced query options dialog gives a 404
2354380
carloa Automation Failure: Search returned results from the SharePoint Source [ACTUAL: 0, EXPECTED: Not 0] (Method:
2358810
jessals Investigating We need a way to resolve a web title from search results
2376343
carloa Query for the top-level of the query fragment web part doesn't include hiddenconstraints
2376391
carloa Expanding a query fragment in the query fragment web part first queries with HiddenConstraints and then without
2386198
grahamc Local Fix Creating an export fails; The given key was not present in the dictionary. at Microsoft.Office.Server.Discovery.Export.GetExportContent()
2387170
grahamc Local Fix SourceFilter in export.xml for an export includes properties that shouldn't have values; export fails
Scenario Health Scenario Blocking Bugs
Status Scenario Task Create a case Create a discovery set Add a mailbox Add a SharePoint site Get the number of items Enter a filter and view the stats Preserve and save the discovery set Preservation works in Exchange Preservation works in SharePoint View auditing report of when content was preserved Create a query Specify a query with a wildcard and proximity Narrow down the query scope View results from Exchange, SharePoint, and file shares Show the hovercard with a preview of the message Open the message Select multiple refinements for SharePoint View the hovercard for SharePoint items Save the query Export the query
White Papers• A big doc of how to do “it”• Guidance, config, setup, best
practices• Write a draft early (before
Beta1)• Give to early customers• Get feedback – hard to
setup? Are the requirements reasonable?
• Add to it when you make a decision that customers should know about
Early Customer Deployments• Beta users• Adoption Program
• High touch:• Weekly synch ups• Prototype testing – Beta 1• Go live – Beta2 – RTM
• Ongoing feedback• Bug finding• Lots of support from engineering• Prove features and fix gaps
Dogfood• Internal use of your features
before official release• Important source of validation and
bug finding• Every product feature should have
a dogfood plan
Secret Guerilla Tactics & Tips• If you don’t have at least one user to talk to – get one• At least do one test with 3 people before release!• Don’t lead users - give tasks, observe, and take notes• Ask users to think aloud• Know your user, you are not the user• When giving tasks – don’t use the phrases in the UI• Don’t take everything you hear at face value• Don’t be afraid to show early designs to real customers• Do projects to validate your features• Use real world data and examples• Measure your health and readiness to ship