Running Successful User Surveys

Running Successful User SurveysSurprises and Lessons Learned

April 6, 2005

Doug StarkSr. Director, Product Management

dstark@medefinance.com510.647.1300

1©2005 MedeFinance. Confidential & Proprietary /

Today’s Discussion

Fast facts about MedeFinance

Our survey process

Key survey questions

Surprises & Next time…

2©2005 MedeFinance. Confidential & Proprietary /

Fast Facts About MedeFinance

Founded By Jim Quist in 1994

Who We Are Web analytic software and client services provider

What We Do Help financial executives improve performance

How We Do It Our Performance On Demand Service1) Adaptive Web Analytics – personalized

2) Instant, on demand IT environment and

3) Client services – deep healthcare domain experience

Headquartered Emeryville, CA

Ownership Private, Profitable, 100% Growth Each of Last 3 Years

Clients Over 400 hospitals…100% Healthcare Focused…

3©2005 MedeFinance. Confidential & Proprietary /

Service MedeFinance Offers

MedeFinance Performance OnDemandTM

On DemandIT

Environment

ClientServices

AdaptiveWeb

Analytics

4©2005 MedeFinance. Confidential & Proprietary /

A Few MedeFinance Clients

5©2005 MedeFinance. Confidential & Proprietary /

Today’s Discussion

Fast facts about MedeFinance

Our survey process

Key survey questions

Surprises & Next time…

6©2005 MedeFinance. Confidential & Proprietary /

Why survey?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

1) Determine the ObjectivesMeasure end-user satisfaction with services & product Measure the company’s perceptionGather input on future product enhancementsDeepen knowledge about our user baseGain experience with direct user surveys

7©2005 MedeFinance. Confidential & Proprietary /

What defines success?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

2) Establish Survey Goals10 – 15 minutes to complete

Response rate of 20 – 25%

Complete in time for roadmap planning

8©2005 MedeFinance. Confidential & Proprietary /

What’s important to know?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

3) Create Survey Questions50 initial questions created for:

User Profile

Services

Product Functionality

Future Functionality

Open Feedback & Optional Self Identification

~75% Quantitative vs. ~25% Qualitative

Cross-functional team wrote questionsProduct Management / Client Services / Account Management / Executives

50 Qs

35 Qs

9©2005 MedeFinance. Confidential & Proprietary /

How should we survey?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

4) Select Survey EngineThere are many options on the market

Key considerations:Advanced AnalysisAnonymous ResponsesBranchingBranding & Easy to UseCost

Email DistributionExport FunctionalityGraphicsResponse TrackingUnsubscribe/Remove

10©2005 MedeFinance. Confidential & Proprietary /

Is it easy to read and use?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

The Survey:

11©2005 MedeFinance. Confidential & Proprietary /

Who should we survey?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

5) Target UsersFrequent to infrequent users

Recent to past users

CFOs, Directors, Managers, Consultants

Gain customer stakeholders’ permission!

Daily50%

Weekly25%

Monthly15%

Quarterly10%

Targeted users logged in: Number of times targeted users logged in:

12©2005 MedeFinance. Confidential & Proprietary /

Why is it important?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

6) Conduct Survey - InvitationPersonalize email invitation from CEO

“As a valued friend and partner of MedeFinance, I trust you have experienced the commitment of our Company to excellence and our goal to make you successful…

The survey starts at:http://www.surveymonkey.com/a...

I look forward to your response by…

Sincerely,Jim Quist, CEO

13©2005 MedeFinance. Confidential & Proprietary /

Did you forget?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

7) Conduct Survey - ReminderAfter 7 days, we experienced a response rate of 25%

Reminder invitation to those who had not responded boosted response rate to 35%!“Last week, Jim Quist, sent an online survey invitation…I want to follow up personally to ensure you receive this invitation…

We truly are shaping the product based on our customer's direct feedback…

I look forward to your response by…

Sincerely,Doug StarkSr. Director, Product Management

14©2005 MedeFinance. Confidential & Proprietary /

What does it all mean?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

8) Analyze ResultsDetermine validity

Compare averages/means

Observe distribution

Correlate results with conjoint analysis

Use common sense…

15©2005 MedeFinance. Confidential & Proprietary /

What is important?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

9) Report ResultsShare key results early!

Don’t make it a science project…

Present results in actionable manner

Publish by account, function, etc…

Provide supporting detail for outliers

Correlate key questions with supporting and/or opposing questions

16©2005 MedeFinance. Confidential & Proprietary /

How to publish the results?

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

Example Results Graphs

0%

1%

14%

58%

27%

Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree

StronglyDisagree

Disagree Neutral StronglyAgree

Not

at

All

Som

eS

uppo

rt W

ell

Eno

ugh

Ext

rem

ely

Wel

l

10. I feel I was ADEQUATELY TRAINED on how to use...

9. H

ow w

ell d

o yo

u K

NO

W H

OW

TO

USE

...?

Agree

ExtremelyDissatisfied

(1)

Dissatisfied(2)

Neutral(3)

Satisfied(4)

ExtremelySatisfied

(5)

Response Rate = n%Response Rate = n%

ExtremelyDissatisfied

(1)

Dissatisfied(2)

Neutral(3)

Satisfied(4)

ExtremelySatisfied

(5)

Response Rate = n%Response Rate = n%

“It would be helpful if…”

“It would be helpful if…”

“We really like…”“We really like…”

n

17©2005 MedeFinance. Confidential & Proprietary /

Get the value!

Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1

10) Take ActionRespondents expect action!

Assign functional leads to create action plans for:Immediate, Tactical, Strategic

Product vs. Service

By Functional Area

Earn a strong future response rate by responding to respondents’ feedback!

18©2005 MedeFinance. Confidential & Proprietary /

Today’s Discussion

Fast facts about MedeFinance

Our survey process

Key survey questions

Surprises & Next time…

19©2005 MedeFinance. Confidential & Proprietary /

Satisfaction

20©2005 MedeFinance. Confidential & Proprietary /

Benefits

21©2005 MedeFinance. Confidential & Proprietary /

Credible Results

22©2005 MedeFinance. Confidential & Proprietary /

Usability & Training

23©2005 MedeFinance. Confidential & Proprietary /

MVQ - Most Valuable Questions

24©2005 MedeFinance. Confidential & Proprietary /

Today’s Discussion

Fast facts about MedeFinance

Our survey process

Key survey questions

Surprises & Next time…

25©2005 MedeFinance. Confidential & Proprietary /

Surprises

The survey was a success!Validation and reality check -significant value can be achieved from a small group n<300Response rate - reminders really do drive response ratesCustomer affinity countsHow much work it really takes

26©2005 MedeFinance. Confidential & Proprietary /

Next time…

Now that we have experience, next time we will…Segment the survey - ask the right questions to the right people

Quality over quantity – perform more analysis with fewer questions. How will the answer be used?

Increase survey group sizeSeriously consider outsourcing!

27©2005 MedeFinance. Confidential & Proprietary /

Go survey!

This was a helpful presentation.Strongly DisagreeDisagreeAverageAgreeStrongly Agree